Dinner plate information display method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a dinner plate information display method and device, electronic equipment and a computer readable medium. A specific embodiment of the method comprises the following steps: acquiring a dinner plate image corresponding to each dinner plate; obtaining each service plate instance segmentation image; the following steps are executed: inputting the service plate instance segmentation image into a feature extraction network in a pre-trained service plate identification model; inputting the dinner plate feature map into a feature processing network in a dinner plate recognition model; inputting the dinner plate full image feature information into a classification processing network in a dinner plate identification model; determining each piece of dinner plate value information corresponding to each dinner plate instance segmentation image; generating dinner plate information; and displaying the dinner plate information. According to the embodiment, the accuracy of identifying the dinner plate image can be improved, and the waste of computing resources can be reduced.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, electronic device, and computer-readable medium for displaying plate information. Background Art

[0002] Plate recognition technology is a technology that recognizes plates based on their images to obtain the category information of the plates. At present, the method commonly used to recognize plates is to build a plate image classification model. Then, the constructed plate image classification model is used to extract features from the pre-acquired plate image. Finally, the category information of the plate is obtained based on the extracted plate features.

[0003] However, when using the above method to identify the plate, the following technical problems often occur: When using the plate image classification model to extract features from plate images, it is easy to be interfered by the background image in the plate image, which can easily lead to errors in the extracted plate features, and then cause the plate image classification model to have low accuracy in classifying and identifying plate images, resulting in the need to re-call computing resources to perform feature extraction and classification on the same plate image, resulting in a waste of computing resources. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure provide a method, device, electronic device, and computer-readable medium for displaying plate information to solve one or more of the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a method for displaying plate information, the method comprising: obtaining a plate image corresponding to each plate; performing instance segmentation processing on the plate image to obtain each plate instance segmentation image; for each of the plate instance segmentation images, performing the following steps: inputting the plate instance segmentation image into a feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image to obtain a plate feature map corresponding to the plate instance segmentation image; inputting the plate feature map into the plate The feature processing network in the recognition model is used to generate full-plate feature information corresponding to the above-mentioned plate feature image; the above-mentioned full-plate feature information is input into the classification processing network in the above-mentioned plate recognition model to obtain the plate category information corresponding to the above-mentioned plate instance segmentation image; based on the obtained respective plate category information and the preset category value information group, the respective plate value information corresponding to the above-mentioned respective plate instance segmentation image is determined; based on the above-mentioned respective plate category information and the above-mentioned respective plate value information, the plate information is generated, wherein the above-mentioned plate information includes the total value information of the plate; and the above-mentioned plate information is displayed.

[0007] In a second aspect, some embodiments of the present disclosure provide a plate information display device, the device comprising: an acquisition unit, configured to acquire a plate image corresponding to each plate; a segmentation unit, configured to perform instance segmentation processing on the plate image to obtain each plate instance segmentation image; an execution unit, configured to perform the following steps for each plate instance segmentation image in the each plate instance segmentation image: input the plate instance segmentation image into a feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image to obtain a plate feature map corresponding to the plate instance segmentation image; input the plate feature map into the plate A feature processing network in the recognition model is used to generate full-plate feature information corresponding to the above-mentioned plate feature image; the above-mentioned full-plate feature information is input into the classification processing network in the above-mentioned plate recognition model to obtain the plate category information corresponding to the above-mentioned plate instance segmentation image; a determination unit is configured to determine the respective plate value information corresponding to the above-mentioned each plate instance segmentation image based on the obtained respective plate category information and the preset category value information group; a generation unit is configured to generate plate information based on the above-mentioned respective plate category information and the above-mentioned respective plate value information, wherein the above-mentioned plate information includes the total value information of the plate; a display unit is configured to display the above-mentioned plate information.

[0008] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0009] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0010] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the plate information display method of the present disclosure, the accuracy of plate image recognition can be improved, and the waste of computing resources can be reduced. Specifically, the reason for the low accuracy of plate image recognition and the waste of computing resources is that when the plate image classification model is used to extract features of the plate image, it is easy to be interfered by the background image in the plate image, which easily leads to errors in the extracted plate features, and then the plate image classification model has low accuracy when classifying and identifying the plate image, resulting in the need to call computing resources again to extract features and classify and identify the same plate image, resulting in a waste of computing resources. Based on this, the plate information display method of some embodiments of the present disclosure first obtains the plate image corresponding to each plate. Thus, the original data to be processed can be obtained. Secondly, the above plate image is subjected to instance segmentation processing to obtain each plate instance segmentation image. Thus, the interference of other factors on the plate image can be reduced, and the accuracy of subsequent classification and identification of the plate image can be improved. Then, for each of the above-mentioned plate instance segmentation images, the following steps are performed: first, the plate instance segmentation image is input into the feature extraction network in the pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image to obtain a plate feature map corresponding to the plate instance segmentation image. Thus, the feature map corresponding to the plate image can be obtained. Secondly, the plate feature map is input into the feature processing network in the plate recognition model to generate plate full-image feature information corresponding to the plate feature map. Thus, the global feature information of the plate image can be obtained. Then, the plate full-image feature information is input into the classification processing network in the plate recognition model to obtain the plate category information corresponding to the plate instance segmentation image. Thus, the category corresponding to the plate image can be obtained. Then, based on the obtained plate category information and the preset category value information group, the plate value information corresponding to each of the plate instance segmentation images is determined. Thus, the value information corresponding to each plate can be determined by the category corresponding to the plate image. Then, based on the above-mentioned information of each plate category and the above-mentioned information of each plate value, the plate information is generated, wherein the above-mentioned plate information includes the total value information of the plate. Thus, the total value information of each plate can be obtained. Finally, the above-mentioned plate information is displayed. Thus, the total value information of the plate can be displayed. Also, because the plate image can be subjected to instance segmentation processing before classification and recognition, the interference of the background image in the plate image on the plate image can be reduced, thereby improving the accuracy of classification and recognition of the plate image.Furthermore, because the global feature information of the plate image can be integrated and then the plate image can be classified and identified according to the features of the plate image, the accuracy of classifying and identifying the plate image can be improved, and the number of times computing resources are called to process the same plate image can be reduced, thereby reducing the waste of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0012] Figure 1 is a flow chart of some embodiments of the method for displaying plate information according to the present disclosure; Figure 2 is a schematic structural diagram of some embodiments of the dinner plate information display device according to the present disclosure; Figure 3 is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure; Figure 4 It is a schematic screenshot of an internal test operation page for displaying plate information according to the plate information display method disclosed in the present invention. DETAILED DESCRIPTION

[0013] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0014] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0015] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0016] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0017] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0018] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0019] Figure 1 The process 100 of some embodiments of the method for displaying plate information according to the present disclosure is shown. The method for displaying plate information comprises the following steps: Step 101, obtaining a plate image corresponding to each plate.

[0020] In some embodiments, the execution subject (e.g., a computing device) of the method for displaying plate information may obtain a plate image corresponding to each plate. The plate image may be an image of each plate. For example, the plate image may be an image captured by a camera before the execution subject identifies each plate with food selected by the user in the restaurant. In practice, the execution subject may obtain the plate image from a preset database. The database may be a database for storing plate images. The execution subject may be a server device that identifies each plate through the plate image and generates plate information corresponding to each plate when the user performs a payment operation in the restaurant. The server device may be a computing device used by restaurant service personnel (e.g., cashier).

[0021] Step 102: perform instance segmentation processing on the dinner plate image to obtain each dinner plate instance segmentation image.

[0022] In some embodiments, the execution subject may perform instance segmentation processing on the dinner plate image to obtain each dinner plate instance segmentation image, wherein the dinner plate instance segmentation image may be an image containing only one dinner plate obtained after performing instance segmentation processing on the dinner plate image.

[0023] In some optional implementations of some embodiments, the execution subject may perform instance segmentation processing on the dinner plate image through the following steps to obtain each dinner plate instance segmentation image: The first step is to preprocess the plate image to obtain a preprocessed plate image. The preprocessed plate image may be a preprocessed plate image. In practice, first, the execution subject may perform denoising on the plate image by Gaussian filtering to obtain a denoised plate image. Then, the resolution of the denoised plate image may be adjusted by resampling technology to obtain a preprocessed plate image.

[0024] The second step is to normalize the pre-processed plate image to obtain a normalized plate image. The normalized plate image may be an image obtained by normalizing the pre-processed plate image. In practice, the execution subject may normalize the pre-processed plate image by an extreme value normalization method to obtain a normalized plate image. The extreme value normalization method may be deviation normalization.

[0025] The third step is to perform feature extraction processing on the above-mentioned normalized plate image to obtain a normalized plate feature map. The above-mentioned normalized plate feature map can be a feature map corresponding to the above-mentioned normalized plate image. In practice, the above-mentioned execution entity can input the above-mentioned normalized plate image into the feature map extraction network of the pre-trained instance segmentation model to obtain the normalized plate feature map. The above-mentioned instance segmentation model can be a neural network model that takes the normalized plate image as input and each instance segmentation mask corresponding to the normalized plate image as output. The above-mentioned instance segmentation model can include four layers.

[0026] The first layer may be a feature map extraction network. The feature map extraction network may be a neural network that takes a normalized plate image as input and a normalized plate feature map as output. For example, the feature map extraction network may be a convolutional neural network.

[0027] The second layer may be a feature map processing layer. The feature map processing layer may include a feature map labeling network and a classification layer. The feature map labeling network may be a neural network that takes the normalized plate feature map as input and each bounding box information corresponding to the normalized plate feature map as output. For example, the feature map labeling network may be a region proposal network. Each of the bounding box information may be information for characterizing the bounding box. The bounding box information may include size information, center point coordinates and offset information corresponding to the bounding box. The size information may be used to characterize the width and height of the bounding box. The center point coordinates may be the coordinates of the center point of the bounding box in the normalized plate feature map. The offset information may be used to characterize the offset of the center point coordinates relative to the center point of the normalized plate feature map. Each of the bounding box information may correspond to a normalized plate feature image region. The normalized plate feature image region may be an image region in the normalized plate feature image.

[0028] The above-mentioned classification layer can be a neural network layer that takes the bounding box information corresponding to the normalized plate feature map output by the above-mentioned feature map labeling network as input, and takes the recognition result corresponding to each bounding box information in the bounding box information as output. For example, the above-mentioned classification layer can be a convolutional layer. Among them, the above-mentioned recognition result can be the relevant information of the normalized plate feature image area corresponding to the bounding box information. The above-mentioned recognition result may include category information and confidence. The above-mentioned category information can be used to characterize the category of the normalized plate feature image area corresponding to the bounding box information. The above-mentioned confidence can be used to characterize the accuracy of the above-mentioned category information. For example, the above-mentioned recognition result can be "the image area is a plate image, 90%".

[0029] The third layer may be a regression layer. The regression layer may be a neural network layer that takes the respective bounding box information corresponding to the normalized plate feature map output by the feature map labeling network and the recognition result corresponding to each bounding box information in the respective bounding box information output by the classification layer as input, and takes the respective adjusted bounding box information corresponding to the normalized plate feature map as output. For example, the regression layer may be a convolutional layer. The adjusted bounding box information may be information used to characterize the adjusted bounding box. The adjusted bounding box information may include adjustment size information, adjustment center point coordinates, and adjustment offset information. The adjusted size information may be used to characterize the width and height of the adjusted bounding box. The adjustment center point coordinates may be the coordinates of the center point of the adjusted bounding box in the normalized plate feature map. The offset information may be used to characterize the offset of the adjustment center point coordinates relative to the center point of the normalized plate feature map. The adjusted bounding box may be a bounding box obtained by enlarging, reducing, translating, etc. the bounding box represented by the bounding box information. Each of the adjusted bounding box information corresponds to the category information and confidence included in the recognition result. Each of the above-mentioned individual adjustment bounding box information corresponds to a normalized dinner plate feature image region.

[0030] The fourth layer may be a mask generation network. The mask generation network may be a neural network that takes the individual adjusted bounding box information corresponding to the normalized plate feature map output by the regression layer as input, and takes the instance segmentation mask of the normalized plate feature image region corresponding to each of the individual adjusted bounding box information as output. For example, the mask generation network may be a fully convolutional network. The instance segmentation mask may be a two-dimensional array corresponding to the normalized plate feature image region. Each element in the two-dimensional array may correspond to a pixel in the normalized plate feature image region. Each element in the two-dimensional array may be used to characterize whether the corresponding pixel belongs to the pixel corresponding to the plate image. As an example, when the pixel belongs to the pixel corresponding to the plate image, the corresponding element in the two-dimensional array may be "1"; when the pixel does not belong to the pixel corresponding to the plate image, the corresponding element in the two-dimensional array may be "0".

[0031] The fourth step is to generate each bounding box information based on the normalized plate feature map. In practice, the execution subject can generate each bounding box information on the normalized plate feature map through the feature map labeling network of the feature map processing layer in the instance segmentation model according to the preset bounding box size information. The bounding box size information can be used to characterize the size of the bounding box.

[0032] In the fifth step, each normalized plate feature image region corresponding to each of the above-mentioned bounding box information is identified and processed to obtain each recognition result. In practice, the above-mentioned execution subject can identify and process each normalized plate feature image region corresponding to each of the above-mentioned bounding box information through the classification layer of the feature map processing layer in the above-mentioned instance segmentation model to obtain each recognition result.

[0033] In step 6, based on the above recognition results, the above bounding box information is adjusted to obtain each adjusted bounding box information. In practice, the execution subject can adjust the above bounding box information through the regression layer in the instance segmentation model to obtain each adjusted bounding box information.

[0034] In the seventh step, based on the confidences corresponding to the above-mentioned adjusted bounding box information, each target bounding box information is determined. The above-mentioned target bounding box information may be the adjusted bounding box information after screening. Each target bounding box information in the above-mentioned target bounding box information corresponds to a normalized plate feature image area.

[0035] In step 8, for each target bounding box information in the above target bounding box information, a corresponding instance segmentation mask is generated based on the normalized plate feature image region corresponding to the target bounding box information. In practice, the execution subject can generate the instance segmentation mask of the normalized plate feature image region corresponding to the target bounding box information through the mask generation network in the instance segmentation model.

[0036] In the ninth step, each target instance segmentation mask is determined based on each confidence corresponding to each generated instance segmentation mask. The target instance segmentation mask may be a screened instance segmentation mask. In practice, in the first step, the execution subject may sort the instance segmentation masks in descending order according to each confidence corresponding to each instance segmentation mask to obtain an instance segmentation mask sequence. In the second step, the instance segmentation mask with the highest confidence corresponding to the instance segmentation mask sequence may be added to the instance segmentation mask list. In the third step, each instance segmentation mask other than the instance segmentation mask with the highest confidence in the instance segmentation mask may be determined as each instance segmentation mask to be processed. In the fourth step, each intersection-and-union ratio of the instance segmentation mask with the highest confidence and each instance segmentation mask to be processed may be generated by a non-maximum suppression algorithm as each mask intersection-and-union ratio. In the fifth step, each mask intersection-and-union ratio greater than a preset mask intersection-and-union ratio threshold in the mask intersection-and-union ratio may be determined as each target mask intersection-and-union ratio. The preset mask intersection-and-union ratio threshold may be a preset value. Here, there is no limitation on the specific setting of the above-mentioned preset mask intersection-over-union ratio threshold. In the sixth step, the instance segmentation mask with the highest confidence and each of the instance segmentation masks corresponding to the intersection-over-union ratios of the above-mentioned target masks can be cleared by a non-maximum suppression algorithm to update the instance segmentation mask sequence. In the seventh step, the first step to the sixth step are re-executed using the updated instance segmentation mask sequence until the updated instance segmentation mask sequence is empty. In the eighth step, each instance segmentation mask in the updated instance segmentation mask list is determined as each target instance segmentation mask.

[0037] The tenth step is to generate each plate instance segmentation image based on the above-mentioned target instance segmentation masks. The plate instance segmentation image may be a plate image obtained after instance segmentation processing. In practice, for each target instance segmentation mask in the above-mentioned target instance segmentation masks, first, the execution subject may determine the image area in the plate image corresponding to the target instance segmentation mask as the target image area. Then, for each element of each element in the target instance segmentation mask, the product of the element and the pixel value corresponding to the element in the target image area may be determined as the plate instance segmentation image pixel value. Then, the determined pixel values ​​of each plate instance segmentation image may be rendered as a plate instance segmentation image through an image processing function. The image processing function may be a Pillow library function.

[0038] In some optional implementations of some embodiments, the execution subject may determine each target bounding box information based on each confidence level corresponding to each adjusted bounding box information through the following steps: In the first step, based on the confidences corresponding to the above-mentioned respective adjusted bounding box information, the above-mentioned respective adjusted bounding box information are sorted to obtain an adjusted bounding box information sequence. In practice, the above-mentioned execution subject can sort the above-mentioned respective adjusted bounding box information in descending order according to the magnitudes of the confidences corresponding to the above-mentioned respective adjusted bounding box information to obtain an adjusted bounding box information sequence.

[0039] In the second step, based on the adjusted bounding box information sequence, the following loop steps are performed: In the first sub-step, the adjusted bounding box information satisfying a preset confidence condition in the adjusted bounding box information sequence is determined as the target adjusted bounding box information, wherein the preset confidence condition may be that the confidence corresponding to the adjusted bounding box information is the highest.

[0040] The second sub-step is to add the target adjustment bounding box information to the target adjustment bounding box information list to update the target adjustment bounding box information list.

[0041] In the third sub-step, each piece of adjusted bounding box information that does not satisfy the above preset reliability condition in the adjusted bounding box information sequence is determined as each piece of adjusted bounding box information to be processed.

[0042] The fourth sub-step is to generate each intersection-and-union ratio based on the target adjustment bounding box information and the respective to-be-processed adjustment bounding box information. In practice, for each to-be-processed adjustment bounding box information in the respective to-be-processed adjustment bounding box information, the execution subject may generate the intersection-and-union ratio between the to-be-processed adjustment bounding box information and the target adjustment bounding box information by a non-maximum suppression algorithm.

[0043] In a fifth sub-step, each I / O ratio that satisfies a preset I / O ratio condition among the above I / O ratios is determined as each target I / O ratio. The preset I / O ratio condition may be that the I / O ratio is greater than a preset I / O ratio threshold. The preset I / O ratio threshold may be a pre-set value. Here, the specific setting of the preset I / O ratio threshold is not limited.

[0044] In a sixth sub-step, based on the adjustment bounding box information sequence, the target adjustment bounding box information and each of the pending adjustment bounding box information corresponding to the above-mentioned target intersection-and-union ratios are cleared to update the adjustment bounding box information sequence. In practice, the execution subject may clear the target adjustment bounding box information and each of the pending adjustment bounding box information corresponding to the above-mentioned target intersection-and-union ratios by a non-maximum suppression algorithm to update the adjustment bounding box information sequence.

[0045] In a seventh sub-step, in response to determining that the updated adjusted bounding box information sequence satisfies a preset loop condition, the above loop step is performed again using the updated adjusted bounding box information sequence, wherein the above preset loop condition may be that the updated adjusted bounding box information sequence is not empty.

[0046] In an eighth sub-step, in response to determining that the updated adjustment bounding box information sequence does not satisfy the preset loop condition, each target adjustment bounding box information in the updated target adjustment bounding box information list is determined as each target bounding box information.

[0047] Step 103, for each dinner plate instance segmentation image in each dinner plate instance segmentation image, perform the following steps: Step 1031 , input the plate instance segmentation image into a feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image to obtain a plate feature map corresponding to the plate instance segmentation image.

[0048] In some embodiments, the execution entity may input the plate instance segmentation image into a feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image to obtain a plate feature map corresponding to the plate instance segmentation image. The plate feature map may be a feature map corresponding to the plate instance segmentation image. The plate recognition model may be a neural network model that takes the plate instance segmentation image as input and takes the plate category information corresponding to the plate instance segmentation image as output. For example, the neural network model may be a convolutional neural network. The plate category information may be information used to characterize the color and shape of the plate. The plate recognition model may include three layers.

[0049] The first layer may be a feature extraction network. The feature extraction network may be a neural network that takes the plate instance segmentation image as input and takes the plate feature map corresponding to the plate instance segmentation image as output. For example, the feature extraction network may be a convolutional neural network. The plate feature map may be a feature map corresponding to the plate instance segmentation image.

[0050] The second layer may be a feature processing network. The feature processing network may be a neural network that takes a plate feature map corresponding to the plate instance segmentation image as input and takes the plate full image feature information corresponding to the plate instance segmentation image as output. For example, the feature processing network may be a graph convolutional neural network. The plate full image feature information may be a feature vector that integrates all features in the plate image.

[0051] The third layer may be a classification processing network. The classification processing network may include a feature vector processing layer and an output layer. The feature vector processing layer may be a fully connected layer that takes the feature information of the entire plate image corresponding to the plate instance segmentation image as input and takes the global updated feature vector corresponding to the plate instance segmentation image as output. The global updated feature vector may be the feature information of the entire plate image processed by the fully connected layer.

[0052] The output layer may include a probability distribution function and an output function. The probability distribution function may be a probability distribution function that takes the global updated feature vector output by the feature vector processing layer as input and takes the probability distribution data of the plate corresponding to the plate instance segmentation image as output. The probability distribution data of the plate may be a probability distribution of the plate category information corresponding to the plate. As an example, the probability distribution data of the plate may be "plate category information: square-white, plate probability data: 90%; plate category information: oval-white, plate probability data: 10%". The probability distribution data of the plate may include each plate probability data. The plate probability data may be the probability that the plate belongs to the plate category information. Each plate probability data in the plate probability data corresponds to the plate category information. The probability distribution function may be a normalized exponential function. The output function may be an argmax function that takes the probability distribution data of the plate as input and takes the plate category information corresponding to the largest plate probability data in the plate probability distribution data as output.

[0053] In some optional implementations of some embodiments, the plate recognition model may be obtained by training the execution subject in the following manner: The first step is to obtain a sample set. Each sample in the sample set includes a plate instance segmentation image and sample plate category information corresponding to the plate instance segmentation image. The sample plate category information can be used to characterize the true category of the plate instance segmentation image.

[0054] In the second step, based on the sample set, perform the following training steps: In the first sub-step, the plate instance segmentation images of at least one sample in the sample set are respectively input into the initial neural network to obtain the plate category information corresponding to each sample in the at least one sample. The structure of the initial neural network can refer to the plate recognition model mentioned above, which will not be described in detail here. The plate category information can be the category corresponding to the plate instance segmentation image.

[0055] The second sub-step is to compare the plate category information corresponding to each sample in the at least one sample with the corresponding sample plate category information. In practice, first, the execution subject may determine the plate probability distribution data corresponding to each sample in the at least one sample as the target plate probability distribution data. Then, the execution subject may generate the probability distribution of the sample plate category information corresponding to each sample in the at least one sample through a normalized exponential function. Finally, the execution subject may generate the loss value between the target plate probability distribution data corresponding to each sample in the at least one sample and the probability distribution of the corresponding sample plate category information through a cross entropy loss function.

[0056] The third sub-step is to determine whether the initial neural network has achieved the preset optimization goal based on the comparison result. As an example, when the loss value between the plate category information corresponding to a sample and the corresponding sample plate category information is less than a preset loss threshold, the sample is determined to be a positive sample. The positive sample can be used to characterize samples that meet the loss value condition. The loss value condition can be that the loss value between the plate category information corresponding to a sample and the corresponding sample plate category information is less than a preset loss threshold. The optimization goal can be that the proportion of positive samples in all samples is greater than a preset accuracy threshold. The loss threshold can be a preset value. Here, there is no limitation on the specific setting of the loss threshold. The accuracy threshold can be a preset value. Here, there is no limitation on the specific setting of the accuracy threshold.

[0057] In a fourth sub-step, in response to determining that the initial neural network achieves the above optimization goal, the initial neural network is determined as a dinner plate recognition model.

[0058] The fifth sub-step is, in response to determining that the initial neural network does not achieve the above optimization goal, adjusting the network parameters of the initial neural network, using unused samples to form a sample set, using the adjusted initial neural network as the initial neural network, and performing the above training step again. In practice, the above execution subject can use the back propagation algorithm (BackPropagation Algorithm, BP algorithm) and the gradient descent method (such as the mini-batch gradient descent algorithm) to adjust the network parameters of the above initial neural network.

[0059] Step 1032: input the plate feature image into a feature processing network in the plate recognition model to generate full plate image feature information corresponding to the plate feature image.

[0060] In some embodiments, the execution subject may input the plate feature map into the feature processing network in the plate recognition model to generate plate full image feature information corresponding to the plate feature map. The plate full image feature information may be a feature vector containing all information in the plate instance segmentation image.

[0061] In the process of adopting technical solutions to solve the above technical problems, the following problems are often accompanied: When the shapes or colors of plates of different categories are similar, the accuracy of the plate image classification model in recognizing the plate image is low, which results in low reliability of the recognized plate category information and the need to call computing resources again to recognize the same plate multiple times, resulting in a waste of computing resources.

[0062] Faced with the above technical problems, we decided to adopt the following solutions: In some optional implementations of some embodiments, the execution subject may input the plate feature map into the feature processing network in the plate recognition model through the following steps to generate full plate feature information corresponding to the plate feature map: The first step is to construct a plate feature connection graph corresponding to the plate feature graph based on the plate feature graph. The plate feature connection graph may be an unweighted undirected graph used to characterize the connection relationship between the pixels in the plate feature graph. Each pixel in the plate feature graph corresponds to a feature vector. Each node in the plate feature connection graph corresponds to node feature information. The node feature information may be a feature vector corresponding to the node. In practice, first, the execution subject may determine each pixel in the plate feature graph as each node. Then, for each of the nodes, the plate feature connection graph may be obtained by connecting the node with each neighbor node corresponding to the node. In practice, for each pixel in the plate feature graph, first, the execution subject may determine the node corresponding to the pixel in the plate feature connection graph as the target node. Then, the feature vector corresponding to the pixel may be determined as the node feature information corresponding to the target node.

[0063] In the second step, for each node feature information in each node feature information included in the above-mentioned plate feature connection graph, the following steps are performed: The first sub-step is to determine each neighbor node characteristic information corresponding to the node characteristic information based on the node characteristic information, wherein the neighbor node characteristic information may be node characteristic information of the neighbor node corresponding to the node characteristic information.

[0064] In the second sub-step, for each piece of neighbor node feature information in the above-mentioned neighbor node feature information, the following steps are performed: Sub-step one, based on the above-mentioned neighbor node feature information, determine the attention coefficient of the above-mentioned node feature information and the above-mentioned neighbor node feature information. In practice, first, the above-mentioned execution subject can determine the product of the above-mentioned node feature information and the preset weight matrix as linear node feature information. Then, the product of the above-mentioned neighbor node feature information and the above-mentioned weight matrix can be determined as linear neighbor node feature information. Then, the above-mentioned linear node feature information and the above-mentioned linear neighbor node feature information can be spliced ​​into linear feature information through a splicing function. Then, the transpose of the preset weight vector can be determined as a transposed weight vector. Finally, the product of the above-mentioned linear feature information and the above-mentioned transposed weight vector can be determined as an attention coefficient. Among them, the above-mentioned weight matrix can be a network parameter used to connect the feature information of each node in the above-mentioned feature processing network. The above-mentioned weight vector can be a network parameter used to control the threshold of the activation function in the above-mentioned feature processing network. The above-mentioned splicing function can be a concat function.

[0065] Sub-step two: determining the product of the attention coefficient and the neighbor node feature information as weighted neighbor node feature information.

[0066] The third sub-step is to determine the sum of the determined weighted neighbor node feature information as the aggregated neighbor node feature information.

[0067] The fourth sub-step is to integrate the node feature information and the aggregated neighbor node feature information to obtain the spliced ​​node feature information. The spliced ​​node feature information may be a feature vector obtained by splicing the node feature information and the aggregated neighbor node feature information. In practice, the execution subject may splice the node feature information and the aggregated neighbor node feature information into the spliced ​​node feature information through a splicing function. The splicing function may be a concat function.

[0068] The fifth sub-step is to determine the product of the splicing node feature information and the first-level parameter as the first-level feature information. The first-level parameter may be a weight matrix in the feature processing network.

[0069] In a sixth sub-step, the sum of the first-level feature information and the second-level parameter is determined as the second-level feature information. The second-level parameter may be a bias vector in the feature processing network.

[0070] The seventh sub-step is to generate node feature information based on the second-level feature information to update the node feature information. In practice, the second-level feature information can be input into an activation function to obtain the node feature information. The activation function can be a linear rectification function.

[0071] In the third step, in response to determining that the feature information of each node included in the above-mentioned plate feature connection graph meets the preset update condition, based on the updated feature information of each node, the feature information of the whole plate image corresponding to the above-mentioned plate feature graph is determined. Among them, the above-mentioned preset update condition can be that the feature information of each node in the above-mentioned plate feature connection graph is updated. The above-mentioned whole plate image feature information can be a feature vector that integrates all feature information in the above-mentioned plate image. In practice, the above-mentioned execution subject can determine the whole plate image feature information corresponding to the above-mentioned plate feature graph through the above-mentioned feature processing network.

[0072] The fourth step is to determine the corresponding memory usage information based on the above-mentioned full plate image feature information. The above-mentioned memory usage information may be the size of the memory space used by the above-mentioned full plate image feature information. In practice, the above-mentioned execution subject may obtain the memory usage information corresponding to the above-mentioned full plate image feature information through the Instrumentation interface.

[0073] Step 5: In response to determining that the memory usage information is less than the preset allocated memory amount, the physical memory corresponding to the full plate image feature information is reduced to adjust the storage resource allocation of the full plate image feature information. The preset allocated memory amount may be a pre-set memory value for storing the full plate image feature information. In practice, first, the execution subject may determine the difference between the memory usage information and the preset allocated memory amount as the allocated amount. Then, the malloc function may be used to allocate the memory in an idle state and having a storage size of the allocated amount to the full plate image feature information to adjust the storage resource allocation of the full plate image feature information.

[0074] Step 6: In response to determining that the memory usage information is greater than the preset allocated memory amount, increase the physical memory corresponding to the full plate image feature information to adjust the storage resource allocation of the full plate image feature information. In practice, first, the execution subject may determine the difference between the memory usage information and the preset allocated memory amount as the released amount. Then, the memory allocated to the full plate image feature information, in an idle state and with a storage size of the released amount, may be allocated to other processes through the malloc function to adjust the storage resource allocation of the full plate image feature information.

[0075] The above technical solution and its related contents, as an inventive point of an embodiment of the present disclosure, solve the problem of "waste of computing resources". The factors that lead to low efficiency of data processing and waste of network resources during marking are often as follows: when the shapes or colors of plates of different categories are similar, the accuracy of the plate image classification model in recognizing the plate image is low. This leads to low reliability of the identified plate category information, resulting in the need to re-call computing resources to recognize the same plate multiple times, resulting in waste of computing resources. If the above factors are solved, the waste of computing resources can be reduced. In order to achieve this effect, the present disclosure first constructs a plate feature connection graph corresponding to the plate feature graph based on the plate feature graph, wherein the plate feature connection graph includes feature information of each node. Thus, a feature graph of the plate can be obtained. Then, for each node feature information in the node feature information included in the plate feature connection graph, the following steps are performed: First, based on the node feature information, determine the feature information of each neighbor node corresponding to the node feature information. Secondly, for each neighbor node feature information in the above-mentioned neighbor node feature information, the following steps are performed: First, based on the above-mentioned neighbor node feature information, the attention coefficient of the above-mentioned node feature information and the above-mentioned neighbor node feature information is determined. Thus, the attention coefficient of the node feature information and the neighbor node feature information can be obtained. Then, the product of the above-mentioned attention coefficient and the above-mentioned neighbor node feature information is determined as the weighted neighbor node feature information. Then, the sum of each determined weighted neighbor node feature information is determined as the aggregated neighbor node feature information. Then, the above-mentioned node feature information and the above-mentioned aggregated neighbor node feature information are integrated and processed to obtain the spliced ​​node feature information. Then, the product of the above-mentioned spliced ​​node feature information and the first-level parameter is determined as the first-level feature information. Then, the sum of the above-mentioned first-level feature information and the second-level parameter is determined as the second-level feature information. Then, based on the above-mentioned second-level feature information, the node feature information is generated to update the above-mentioned node feature information. Thus, the above-mentioned node feature information can be made to contain the features of each neighbor node feature information by iteratively convolving the neighbor node feature information corresponding to the above-mentioned node feature information. Then, in response to determining that the feature information of each node included in the above-mentioned plate feature connection graph meets the preset update condition, based on the updated feature information of each node, the feature information of the whole plate image corresponding to the above-mentioned plate feature graph is determined. Thus, the feature vector of the whole plate image can be obtained. Then, based on the above-mentioned full plate image feature information, the corresponding memory usage information is determined. Thus, the size of the memory space occupied by the full plate image feature information can be obtained. Then, in response to determining that the above-mentioned memory usage information is less than the preset allocated memory amount, the physical memory corresponding to the above-mentioned full plate image feature information is reduced to adjust the storage resource allocation of the above-mentioned full plate image feature information. Thus, the waste of storage resources can be reduced.Finally, in response to determining that the memory usage information is greater than the preset allocated memory amount, the physical memory corresponding to the full plate image feature information is increased to adjust the storage resource allocation of the full plate image feature information. In this way, data loss caused by insufficient storage space can be reduced. Also, when processing the plate image, the feature map of the plate image can be obtained first, and then the features of the plate image are further processed according to the feature map to obtain the global feature vector of the plate image, and then the global feature vector is further processed by the fully connected layer, and then the category corresponding to the plate image is generated according to the feature vector output by the fully connected layer, so that the local features of the plate can be better extracted, and the plates with similar shapes and colors can be better distinguished by the local features, thereby improving the accuracy of the generated plate category information, reducing the number of times the computing resources are called to identify the same plate, and thereby reducing the waste of computing resources.

[0076] Step 1033 , input the feature information of the whole plate image into the classification processing network in the plate recognition model to obtain the plate category information corresponding to the plate instance segmentation image.

[0077] In some embodiments, the execution entity may input the feature information of the entire plate image into a classification processing network in the plate recognition model to obtain the plate category information corresponding to the plate instance segmentation image.

[0078] In some optional implementations of some embodiments, the execution subject may input the feature information of the whole plate image into the classification processing network in the plate recognition model through the following steps to obtain the plate category information corresponding to the plate instance segmentation image: The first step is to generate the plate probability distribution data corresponding to the plate full image feature information based on the plate full image feature information. In practice, the execution subject can generate the plate probability distribution data corresponding to the plate full image feature information through the feature vector processing layer and the probability distribution function in the output layer in the classification processing network.

[0079] In the second step, each plate probability data included in the plate probability distribution data is determined as each plate category probability data.

[0080] In the third step, the plate category probability data satisfying the preset data condition among the plate category probability data are determined as the target probability data, wherein the preset data condition may be that the plate category probability data is the largest.

[0081] In the fourth step, the dinner plate category information corresponding to the target probability data is determined as the dinner plate category information corresponding to the dinner plate instance segmentation image.

[0082] Step 104 : determining each plate value information corresponding to each plate instance segmented image based on the obtained plate category information and the preset category value information group.

[0083] In some embodiments, the execution subject may determine the value information of each plate corresponding to each plate instance segmentation image based on the obtained plate category information and the preset category value information group. The category value information in the category value information group may be used to characterize the value corresponding to each category of plates. Each category value information in the category value information group may include plate category information and the value of the plate category information. The plate value information may be used to characterize the value of the plate corresponding to the plate instance segmentation image. In practice, for each plate category information in the plate category information, first, the execution subject may use a search function to search for the plate category information that is the same as the plate category information in the category value information group as the target plate category information. Then, the value corresponding to the target plate category information is determined as the target value. Then, the target value is determined as the plate value information of the plate instance segmentation image corresponding to the plate category information. For example, the search function may be a SEARCH function. As an example: when the category value information group is "square-white: 3 yuan; round-white: 2 yuan", and the plate category information corresponding to the plate instance segmentation image is "square-white", the plate value information corresponding to the plate instance segmentation image is "3 yuan".

[0084] Step 105 : Generate meal plate information based on the meal plate category information and the meal plate value information.

[0085] In some embodiments, the execution entity may generate the plate information based on the plate category information and the plate value information. The plate information may be the relevant information corresponding to the plate. The plate information may include the plate total value information. The plate total value information may be used to represent the sum of the plate value information. In practice, the execution entity may combine the plate category information, the plate value information and the plate total value information into the plate information.

[0086] Step 106, displaying the plate information.

[0087] In some embodiments, the execution entity may display the plate information.

[0088] For example, Figure 4 It is a schematic screenshot of an internal test operation page for displaying plate information according to the plate information display method disclosed in the present invention.

[0089] Figure 4"November 4, 2024 17:06:59" may be the current time. "Dinner cashier" may be the name of the task currently executed by the above-mentioned execution subject. "Clear machine" may be a control for clearing data of the above-mentioned execution subject. "Order" may be a control for displaying all orders. "Plate" may be a control for displaying the above-mentioned category value information group. The setting control may be a control for zooming in or out of the page. The homepage control may be a control for exiting the current page and returning to the homepage. Figure 4 The "plate identification information" on the left side can be the plate category information, plate value information and confidence level corresponding to each plate. The confidence level can be used to characterize the accuracy of the above plate category information. Figure 4 Take "oval-white 1 yuan [0.90]" in as an example. Among them, "oval-white" is the category information of the plate. "1 yuan" is the value information of the plate. "[0.90]" is the confidence level. "3 servings in total, 6 yuan in total" can be information used to characterize the number of each identified plate and the total value of each plate. "Server" can be used to display whether the above-mentioned execution entity is connected to the server. "Payment service" can be the name of the task performed by the above-mentioned execution entity. The order information can be relevant information of all orders completed so far. The "current number of orders" in the order information can be the number of all orders completed so far. The "total amount" in the order information can be the sum of the values ​​of all completed orders. Figure 4 On the right side, the "total amount" can be the total value of each plate on the left. The "identification amount" can be the sum of the value information corresponding to each plate when each plate is identified. The "pricing amount" can be the value manually entered by the user through the operation keyboard in the "data selection" module. The "discount amount" can be the value that can be discounted. "Data selection" can be a module for correcting the "identification amount". The operation keyboard in "data selection" can be various controls for modifying the "identification amount" and "discount amount". "No discount" can be used to display each plate that is not currently discounted. "Circle-white" can be used to display the information corresponding to the plates that are circular in shape and white in color. "Square-white" can be used to display the information corresponding to the plates that are square in shape and white in color. The keyboard control can be a control for calling out the keyboard. "Re-identification" can be a control for Figure 4A control for re-identifying the plates on the left. "Merge Payment" may be a control for merging two or more orders into one payment. "Automatic Payment" may be a control for automatically completing payment through mobile payment or other means. "Cash Payment" may be a control for completing payment through paper value (e.g., cash). "Cooperative Payment" may be a control for completing payment through both the above-mentioned "Automatic Payment" and the above-mentioned "Cash Payment". "Confirm Payment" may be a control for confirming payment. "Suspend Order" may be a control for saving an order for subsequent processing. "Cancel Payment" may be a control for canceling payment.

[0090] Optionally, after step 103, the execution entity may further perform the following steps: The first step is to store the above-mentioned information of each plate category in a preset database, wherein the above-mentioned preset database may be a database for storing the information of the plate category.

[0091] The second step is to obtain the information of each plate category within a preset time range as the classification information of each plate. The preset time range may be within a preset time from the current time. The preset time may be one year. In practice, the execution entity may obtain the information of each plate category within the preset time range from the preset database.

[0092] The third step is to classify the above-mentioned plate classification information to obtain a plate classification information group set. Each plate classification information group in the above-mentioned plate classification information group set corresponds to the same plate classification information. In practice, the plate classification information with the same category name in the above-mentioned plate classification information can be determined as a plate classification information group to obtain a plate classification information group set.

[0093] Step 4: for each meal plate classification information group in the above meal plate classification information group set, perform the following steps: The first sub-step is to generate the to-be-transported item information corresponding to the above-mentioned plate classification information group based on the above-mentioned plate classification information group and the preset plate item information. The above-mentioned plate item information may be the relevant information of the food placed on the plates of each category. The above-mentioned plate item information may include the item name and the item weight. The above-mentioned to-be-transported item information may be used to characterize the item name and the total item weight corresponding to the items to be transported.

[0094] The second sub-step is to generate the plate item transportation position information based on the plate classification information group and the preset plate placement information. The plate placement information can be used to characterize the positions where plates of different categories should be placed. The plate placement information can include each plate classification information and the position corresponding to each plate classification information in the plate classification information. The plate item transportation position information can be used to characterize the position where the items corresponding to the items to be transported should be transported. In practice, for each plate classification information in the plate classification information group, first, the execution subject can use a search function to search for the plate classification information that is the same as the plate classification information in the plate placement information as the target plate classification information. Then, the position corresponding to the target plate classification information can be determined as the plate item transportation position. Finally, the determined plate item transportation positions can be combined into the plate item transportation position information. For example, the search function can be a SEARCH function.

[0095] The third sub-step is to control the corresponding item transport device to transport the items corresponding to the item information to be transported to the target location corresponding to the item transport location information of the meal tray based on the meal tray classification information group. The item transport device may be a device for transporting items. For example, the item transport device may be an unmanned vehicle. The target location may be the location represented by the item transport location information of the meal tray.

[0096] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the plate information display method of the present disclosure, the accuracy of plate image recognition can be improved, and the waste of computing resources can be reduced. Specifically, the reason for the low accuracy of plate image recognition and the waste of computing resources is that when the plate image classification model is used to extract features of the plate image, it is easy to be interfered by the background image in the plate image, which easily leads to errors in the extracted plate features, and then the plate image classification model has low accuracy when classifying and identifying the plate image, resulting in the need to call computing resources again to extract features and classify and identify the same plate image, resulting in a waste of computing resources. Based on this, the plate information display method of some embodiments of the present disclosure first obtains the plate image corresponding to each plate. Thus, the original data to be processed can be obtained. Secondly, the above plate image is subjected to instance segmentation processing to obtain each plate instance segmentation image. Thus, the interference of other factors on the plate image can be reduced, and the accuracy of subsequent classification and identification of the plate image can be improved. Then, for each of the above-mentioned plate instance segmentation images, the following steps are performed: first, the plate instance segmentation image is input into the feature extraction network in the pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image to obtain a plate feature map corresponding to the plate instance segmentation image. Thus, the feature map corresponding to the plate image can be obtained. Secondly, the plate feature map is input into the feature processing network in the plate recognition model to generate plate full-image feature information corresponding to the plate feature map. Thus, the global feature information of the plate image can be obtained. Then, the plate full-image feature information is input into the classification processing network in the plate recognition model to obtain the plate category information corresponding to the plate instance segmentation image. Thus, the category corresponding to the plate image can be obtained. Then, based on the obtained plate category information and the preset category value information group, the plate value information corresponding to each of the plate instance segmentation images is determined. Thus, the value information corresponding to each plate can be determined by the category corresponding to the plate image. Then, based on the above-mentioned information of each plate category and the above-mentioned information of each plate value, the plate information is generated, wherein the above-mentioned plate information includes the total value information of the plate. Thus, the total value information of each plate can be obtained. Finally, the above-mentioned plate information is displayed. Thus, the total value information of the plate can be displayed. Also, because the plate image can be subjected to instance segmentation processing before classification and recognition, the interference of the background image in the plate image on the plate image can be reduced, thereby improving the accuracy of classification and recognition of the plate image.Furthermore, because the global feature information of the plate image can be integrated and then the plate image can be classified and identified according to the features of the plate image, the accuracy of classifying and identifying the plate image can be improved, and the number of times computing resources are called to process the same plate image can be reduced, thereby reducing the waste of computing resources.

[0097] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a plate information display device. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0098] like Figure 2 As shown, the plate information display device 200 of some embodiments includes: an acquisition unit 201, a segmentation unit 202, an execution unit 203, a determination unit 204, a generation unit 205 and a display unit 206. The acquisition unit 201 is configured to acquire a plate image corresponding to each plate; the segmentation unit 202 is configured to perform instance segmentation processing on the plate image to obtain each plate instance segmentation image; the execution unit 203 is configured to perform the following steps for each plate instance segmentation image in the plate instance segmentation images: input the plate instance segmentation image into a feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image to obtain a plate feature map corresponding to the plate instance segmentation image; input the plate feature map into a feature processing network in the plate recognition model to generate a plate feature map; The whole plate image feature information corresponding to the above plate feature image is obtained; the whole plate image feature information is input into the classification processing network in the above plate recognition model to obtain the plate category information corresponding to the above plate instance segmentation image; the determination unit 204 is configured to determine the plate value information corresponding to the above plate instance segmentation image based on the obtained plate category information and the preset category value information group; the generation unit 205 is configured to generate the plate information based on the above plate category information and the above plate value information, wherein the plate information includes the total value information of the plate; the display unit 206 is configured to display the above plate information.

[0099] It is understood that the units described in the device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 200 and the units included therein, and will not be described in detail here.

[0100] Reference below Figure 3, which shows a structural schematic diagram of an electronic device (such as a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0101] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0102] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0103] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0104] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0105] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0106] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the plate image corresponding to each plate; performs instance segmentation processing on the plate image to obtain each plate instance segmentation image; for each of the plate instance segmentation images, performs the following steps: inputs the plate instance segmentation image into a feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image to obtain a plate feature map corresponding to the plate instance segmentation image; and extracts the plate feature map corresponding to the plate instance segmentation image. The image is input into the feature processing network in the above-mentioned plate recognition model to generate full-plate feature information corresponding to the above-mentioned plate feature image; the full-plate feature information is input into the classification processing network in the above-mentioned plate recognition model to obtain the plate category information corresponding to the above-mentioned plate instance segmentation image; based on the obtained respective plate category information and the preset category value information group, the respective plate value information corresponding to the above-mentioned respective plate instance segmentation image is determined; based on the respective plate category information and the respective plate value information, the plate information is generated, wherein the above-mentioned plate information includes the total value information of the plate; and the above-mentioned plate information is displayed.

[0107] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, a segmentation unit, an execution unit, a determination unit, a generation unit, and a display unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring plate images corresponding to each plate".

[0110] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that may be used include, without limitation, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0111] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for displaying plate information, comprising: Get the plate image corresponding to each plate; Performing instance segmentation processing on the dinner plate image to obtain each dinner plate instance segmentation image; For each of the dinner plate instance segmentation images, the following steps are performed: Inputting the meal plate instance segmentation image into a feature extraction network in a pre-trained meal plate recognition model to perform feature extraction processing on the meal plate instance segmentation image to obtain a meal plate feature map corresponding to the meal plate instance segmentation image; Inputting the plate feature image into a feature processing network in the plate recognition model to generate full-plate feature information corresponding to the plate feature image; Inputting the feature information of the whole plate image into the classification processing network in the plate recognition model to obtain the plate category information corresponding to the plate instance segmentation image; Based on the obtained category information of each plate and the preset category value information group, determining the value information of each plate corresponding to each plate instance segmented image; Generate meal plate information based on the meal plate category information and the meal plate value information, wherein the meal plate information includes meal plate total value information; The meal plate information is displayed.

2. The method according to claim 1, wherein: The plate recognition model is trained in the following way: Acquire a sample set, wherein each sample in the sample set includes a meal plate instance segmentation image and sample meal plate category information corresponding to the meal plate instance segmentation image; Based on the sample set, the following training steps are performed: Inputting the plate instance segmentation images of at least one sample in the sample set into the initial neural network respectively to obtain the plate category information corresponding to each sample in the at least one sample, wherein the initial neural network includes a feature extraction network, a feature processing network and a classification processing network; comparing the plate category information corresponding to each sample of the at least one sample with the corresponding sample plate category information; Determine whether the initial neural network reaches the preset optimization goal according to the comparison result; In response to determining that the initial neural network achieves the optimization goal, determining the initial neural network as a plate recognition model; In response to determining that the initial neural network does not achieve the optimization goal, the network parameters of the initial neural network are adjusted, and unused samples are used to form a sample set, and the adjusted initial neural network is used as the initial neural network to perform the training step again.

3. The method according to claim 1, wherein: The step of inputting the full plate image feature information into the classification processing network in the plate recognition model to obtain the plate category information corresponding to the plate instance segmentation image includes: Based on the plate full image feature information, generating plate probability distribution data corresponding to the plate full image feature information, wherein the plate probability distribution data includes individual plate probability data, and each plate probability data in the individual plate probability data corresponds to plate category information; Determine each plate probability data included in the plate probability distribution data as each plate category probability data; Determining the plate category probability data satisfying the preset data condition among the plate category probability data as target probability data; The dinner plate category information corresponding to the target probability data is determined as the dinner plate category information corresponding to the dinner plate instance segmentation image.

4. The method according to claim 1, wherein: The method further comprises: Storing the information of each plate category in a preset database; Acquire the category information of each meal plate within a preset time range as the classification information of each meal plate; Classifying and processing each of the plate classification information to obtain a plate classification information group set, wherein each plate classification information group in the plate classification information group set corresponds to the same plate classification information; For each meal plate classification information group in the meal plate classification information group set, the following steps are performed: Based on the meal tray classification information group and preset meal tray item information, generating information of items to be transported corresponding to the meal tray classification information group; Generate the meal tray item transportation location information based on the meal tray classification information group and the preset meal tray placement information; Based on the meal tray classification information group, the corresponding article transporting equipment is controlled to transport the article corresponding to the to-be-transported article information to the target position corresponding to the meal tray article transporting position information.

5. The method according to claim 1, wherein: The performing instance segmentation processing on the dinner plate image to obtain each dinner plate instance segmentation image comprises: Preprocessing the dinner plate image to obtain a preprocessed dinner plate image; Normalizing the preprocessed plate image to obtain a normalized plate image; Performing feature extraction processing on the normalized dinner plate image to obtain a normalized dinner plate feature map; Based on the normalized plate feature map, generating each bounding box information, wherein each bounding box information in the each bounding box information corresponds to a normalized plate feature image region; Performing recognition processing on each normalized plate feature image region corresponding to each of the boundary box information to obtain each recognition result; Based on the respective recognition results, adjusting the respective bounding box information to obtain respective adjusted bounding box information, wherein each adjusted bounding box information in the respective adjusted bounding box information corresponds to a confidence level; Determine each target bounding box information based on each confidence level corresponding to each adjusted bounding box information, wherein each target bounding box information in the each target bounding box information corresponds to a normalized dinner plate feature image region; For each target bounding box information in the target bounding box information, generating a corresponding instance segmentation mask based on a normalized plate feature image region corresponding to the target bounding box information, wherein the instance segmentation mask corresponds to a confidence level; Determine each target instance segmentation mask based on each confidence level corresponding to each generated instance segmentation mask; Based on the segmentation masks of the target instances, segmentation images of the dinner plates are generated.

6. The method according to claim 5, wherein: The determining each target bounding box information based on each confidence level corresponding to each adjusted bounding box information includes: Based on the confidence levels corresponding to the respective pieces of adjusted bounding box information, the respective pieces of adjusted bounding box information are sorted to obtain an adjusted bounding box information sequence; Based on the sequence of adjusted bounding box information, the following loop steps are performed: Determine the adjusted bounding box information that meets a preset confidence condition in the adjusted bounding box information sequence as the target adjusted bounding box information; Adding the target adjustment bounding box information to a target adjustment bounding box information list to update the target adjustment bounding box information list; determining each piece of adjusted bounding box information that does not satisfy the preset confidence condition in the adjusted bounding box information sequence as each piece of adjusted bounding box information to be processed; Generate each intersection-over-union ratio based on the target adjusted bounding box information and each to-be-processed adjusted bounding box information; Determine each I / O ratio that satisfies a preset I / O ratio condition among the I / O ratios as each target I / O ratio; Based on the adjustment bounding box information sequence, clearing the target adjustment bounding box information and each to-be-processed adjustment bounding box information corresponding to each target intersection-over-union ratio, so as to update the adjustment bounding box information sequence; In response to determining that the updated adjusted bounding box information sequence satisfies a preset loop condition, executing the loop step again using the updated adjusted bounding box information sequence; In response to determining that the updated adjustment bounding box information sequence does not satisfy the preset loop condition, each target adjustment bounding box information in the updated target adjustment bounding box information list is determined as each target bounding box information.

7. A plate information display device, comprising: an acquisition unit, configured to acquire a dinner plate image corresponding to each dinner plate; a segmentation unit configured to perform instance segmentation processing on the dinner plate image to obtain each dinner plate instance segmentation image; The execution unit is configured to perform the following steps for each of the plate instance segmentation images: input the plate instance segmentation image into a feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image to obtain a plate feature map corresponding to the plate instance segmentation image; input the plate feature map into a feature processing network in the plate recognition model to generate full plate image feature information corresponding to the plate feature map; input the full plate image feature information into a classification processing network in the plate recognition model to obtain plate category information corresponding to the plate instance segmentation image; a determination unit configured to determine each plate value information corresponding to each plate instance segmented image based on the obtained each plate category information and a preset category value information group; A generating unit configured to generate plate information based on the plate category information and the plate value information, wherein the plate information includes plate total value information; The display unit is configured to display the plate information.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images