Marked menu recognition method and device based on indentation detection

Abstract

The application provides a marked menu recognition method and device based on indentation detection. The marked menu recognition method based on indentation detection comprises the following steps: obtaining a marked menu image; performing image segmentation processing based on indentations in the marked menu image to obtain an indentation image and a menu background image; framing each indentation region in the indentation image; detecting and recognizing text in the indentation region and the menu background image based on a text detection and recognition model to obtain the meaning corresponding to each indentation and each dish name in the menu background image, wherein the meaning corresponding to the indentation comprises at least one of a text meaning, a numerical meaning and a special symbol meaning; calculating the position information of the text in each indentation region and the menu background image by using an intersection over union algorithm to obtain the dish name corresponding to each indentation region; and outputting the dish name corresponding to each indentation region and the number of dishes. The method improves the accuracy of menu marking content recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a marked menu recognition method and device based on indentation detection. BACKGROUND

[0002] With the rapid development of China's economy, people's living standards have been continuously improved, but some relatively cumbersome work is still completed by manual work. In the catering service industry, customers can use mobile phones to scan codes to order, but the customers in the catering industry are diverse, and among the many people, there are customers who cannot use mobile phones to scan codes to order, so restaurants all retain the traditional menu ordering method. In some restaurants with many single products, such as hot pot restaurants, it takes a long time for employees to enter the menu ordered by customers into the system each time, and the work efficiency is low.

[0003] The restaurant robot in the prior art that enters the menu through text recognition solves the problem of low work efficiency of manual menu entry, but since the styles of menus are various, the marks made by customers are various or there is a problem of unclear marks, and in the process of using machine recognition, recognition errors are prone to occur.

[0004] Therefore, the accuracy of menu mark content recognition is low. SUMMARY

[0005] Therefore, the present application provides a marked menu recognition method and device based on indentation detection to solve the above problems.

[0006] According to a first aspect of the present application, a marked menu recognition method based on indentation detection is provided, characterized in that it comprises: acquiring a marked menu image; performing image segmentation processing based on indentations in the marked menu image to obtain an indentation image and a menu background image; framing each indentation region in the indentation image; detecting and recognizing the text in the indentation region and the menu background image based on a text detection and recognition model to obtain the meaning corresponding to each indentation and each dish name in the menu background image, wherein the meaning corresponding to the indentation includes at least one of a text meaning, a digital meaning and a special symbol meaning; calculating the position information of the text in each indentation region and the menu background image by an intersection over union algorithm to obtain the dish name corresponding to each indentation region; and outputting the dish name corresponding to each indentation region and the number of dishes.

[0007] In another implementation manner of the present application, the marked menu recognition method based on indentation detection further comprises: monitoring whether a menu is placed under the camera equipment through a sensor; if it is monitored that the menu is placed under the camera equipment, the camera equipment and the light supplementing equipment automatically start to work; and acquiring the marked menu image comprises: based on the camera equipment and the light supplementing equipment, collecting the marked menu image.

[0008] In another implementation manner of the present application, the image segmentation processing based on the indentation in the marked menu image obtains the indentation image and the menu background image, comprising: performing recognition processing on the marked menu image based on the indentation recognition technology of three-dimensional scanning to obtain an indentation recognition result; and performing image segmentation processing according to the indentation recognition result to obtain the indentation image and the menu background image.

[0009] In another implementation manner of the present application, the marked menu recognition method based on indentation detection further comprises: performing character recognition processing on the indentation image to obtain a character recognition result, wherein the character recognition result comprises at least one of text, number and special symbol.

[0010] In another implementation manner of the present application, the frame selection of each indentation region in the indentation image comprises: using different colors to frame each indentation region corresponding to text, number and special symbol in the indentation image.

[0011] In another implementation manner of the present application, the position information of each indentation region and the text in the menu background image is calculated by the intersection over union algorithm to obtain the corresponding dish name of each indentation region, comprising: calculating the position information of each indentation region and the text in the menu background image by the intersection over union algorithm, and the formula of the intersection over union algorithm is as follows:

[0012]

[0013] Wherein, IOU is the intersection over union, B1 and B2 are the regions corresponding to the indentation region and the text in the menu background image respectively; based on the position information, the corresponding dish name of each indentation region is determined.

[0014] In another implementation manner of the present application, the dish name and the dish quantity corresponding to each indentation region are output, comprising: determining the dish quantity based on the number recognition result in the indentation image; transmitting the dish name and the dish quantity to the kitchen background; and printing the dish name and the dish quantity by the kitchen background.

[0015] According to a second aspect of the present application, a marked menu recognition device based on indentation detection is provided, characterized in that it comprises: an image acquisition module, configured to acquire a marked menu image; an image processing module, configured to perform image segmentation processing based on indentations in the marked menu image to obtain an indentation image and a menu background image; frame each indentation region in the indentation image; detect and recognize text in the indentation region and the menu background image based on a text detection and recognition model to obtain the meaning corresponding to each indentation and each dish name in the menu background image, wherein the meaning corresponding to the indentation comprises at least one of a text meaning, a numerical meaning, and a special symbol meaning; calculate the position information of the text in each indentation region and the menu background image by an intersection over union algorithm to obtain the dish name corresponding to each indentation region; and a result output module, configured to output the dish name corresponding to each indentation region and the dish quantity.

[0016] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of any of the above marked menu recognition methods based on indentation detection when executing the computer program.

[0017] According to a fourth aspect of the present application, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program implements the steps of any of the above marked menu recognition methods based on indentation detection when executed by a processor.

[0018] In the marked menu recognition method based on indentation detection of the present application, the image of the indentation region in the marked menu image and the menu background image are extracted separately, the indentation image is subjected to character meaning analysis, the obtained character meaning is more accurate, the menu background image is extracted from the marked menu image, text recognition is performed on the extracted menu background image to obtain each dish name, the subsequent indentation region and dish name can be well corresponded, the position information of the text in each indentation region and the menu background image is calculated by an intersection over union algorithm, the obtained dish name corresponding to each indentation region is more accurate, the working efficiency is ensured, and the accuracy of menu marking content recognition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. The advantages and benefits of the solutions will become clear to those skilled in the art by reading the following detailed description of the embodiments. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered as limiting the present application. In the drawings:

[0020] Figure 1A step flow chart of a marked menu recognition method based on indentation detection for an embodiment of the present application.

[0021] Figure 2 A flowchart of a text recognition technology PP-OCRv3 (text recognition technology) employed for another embodiment of the present application.

[0022] Figure 3 A structural block diagram of a marked menu recognition device based on indentation detection for another embodiment of the present application.

[0023] Figure 4 A structural diagram of an electronic device for another embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and in detail below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art should belong to the scope of protection of the embodiments of the present application.

[0025] Figure 1 A step flow chart of a marked menu recognition method based on indentation detection provided by an embodiment of the present application is shown in FIG. 1, and the present embodiment mainly includes the following steps: Figure 1

[0026] S101, obtaining a marked menu image.

[0027] Illustratively, a user marks a paper menu according to requirements, and after marking, the marked menu image can be obtained by a delivery robot of a restaurant.

[0028] S102, performing image segmentation processing based on indentations in the marked menu image to obtain an indentation image and a menu background image.

[0029] Illustratively, an indentation recognition technology is used to determine the indentation region in the marked menu image, and an image segmentation technology is used to perform image segmentation processing on the indentation region and the background region to obtain the indentation image and the menu background image, respectively.

[0030] S103, boxing each indentation region in the indentation image.

[0031] Illustratively, according to different user requirements, different colors and shapes of boxes can be set to box each indentation region in the indentation image.

[0032] ​S104, detecting and identifying the text in the indentation area and the menu background image based on the text detection and identification model, obtaining the meaning corresponding to each indentation and each dish name in the menu background image, wherein the meaning corresponding to the indentation includes at least one of text meaning, numerical meaning and special symbol meaning.

[0033] Exemplarily, when detecting and identifying the characters in the indentation area and the text in the menu background image, the PP-OCRv3 (text recognition) technology can be used, such as Figure 2 As shown in the figure, the PP-OCRv3 (text recognition) technology mainly includes text input, text detection, text box correction, text recognition and text output.

[0034] It should be understood that the text detection process includes inputting the menu background image into the processor, and the processor detects the text in the menu background image. The text box correction process includes that the position of the text box in the detected menu background image may be inclined, and the horizontal correction of the text box is needed by training the text direction classifier, which is convenient for the subsequent text recognition. Text recognition: text recognition of the corrected text box in the menu background image.

[0035] The text recognition process in the indentation image is the same as the text recognition process of the menu background image, and the PP-OCRv3 (text recognition) technology can be used to recognize the text meaning, numerical meaning and special symbol meaning contained in the indentation image.

[0036] When recognizing the text in the menu image, the PP-OCRv3 technology framework is used. This technology can accurately recognize the text content in complex environment, and the recognition effect is good. This technology can be trained combined with the data of the menu, combined with the project requirements, deployed on the delivery robot, which can effectively recognize the menu text and improve the recognition accuracy.

[0037] S105, calculating the position information of the text in each indentation area and the menu background image by the intersection over union algorithm, obtaining the dish name corresponding to each indentation area.

[0038] Exemplarily, the position information of the text in each indentation area and the menu background image is calculated by the intersection over union algorithm, and the formula of the intersection over union algorithm is as follows:

[0039]

[0040] Wherein, IOU is the intersection over union, B1 and B2 are the regions corresponding to the text in the indentation area and the menu background image respectively. According to the calculation result, the dish name corresponding to each indentation area is determined.

[0041] S106, outputting the dish name and the number of dishes corresponding to each indentation area.

[0042] Exemplarily, the dish name and the dish quantity corresponding to each indentation area can be output to the terminal by the dish delivery robot in a fixed format, and then transmitted to the kitchen background equipment for printing.

[0043] In the marked menu recognition method based on indentation detection, the image of the indentation area in the marked menu image and the menu background image are extracted separately, character meaning analysis is performed on the indentation image, the obtained character meaning is more accurate, the menu background image is extracted from the marked menu image, text recognition is performed on the extracted menu background image, and each dish name is obtained, so that the indentation area and the dish name can be well corresponded in the subsequent process, the position information of each indentation area and the text of the menu background image is calculated through the intersection-over-union algorithm, and the dish name corresponding to the obtained indentation area is more accurate, while the working efficiency is ensured, and the accuracy of the marked menu content recognition is improved.

[0044] In another implementation mode of the present application, the marked menu recognition method based on indentation detection further includes: monitoring whether a menu is placed under the camera equipment through a sensor; if it is monitored that the menu is placed under the camera equipment, the camera equipment and the light supplement equipment automatically start working; and obtaining the marked menu image, including: based on the camera equipment and the light supplement equipment, collecting the marked menu image.

[0045] Exemplarily, an infrared sensing device can be installed on the dish delivery robot, which is used to sense whether there is an object under the camera equipment, if there is an object, the light supplement lamp and the camera equipment are automatically turned on to collect the marked menu data. The infrared sensing technology can achieve the effect of energy saving and environmental protection, the camera equipment adopts the light supplement method to reduce the influence of the environment on the picture features when extracting the picture, and the marked menu is shot by the camera equipment and imported into the processor. The infrared sensing device, the camera equipment and the processor can be arranged on the dish delivery robot.

[0046] In another implementation mode of the present application, the image segmentation processing is performed based on the indentation in the marked menu image to obtain the indentation image and the menu background image, including: performing recognition processing on the marked menu image based on the indentation recognition technology of three-dimensional scanning to obtain the indentation recognition result; and performing image segmentation processing according to the indentation recognition result to obtain the indentation image and the menu background image.

[0047] Exemplarily, the marked menu is processed in the processor through the handwriting indentation recognition technology of three-dimensional scanning to separate the indentation area and the original menu background area, and form a separate indentation image. The handwriting indentation recognition technology of three-dimensional scanning can make the indentation appear, and then obtain the handwriting indentation image. By using this technology to extract the handwriting indentation information, the interference of the menu background can be avoided when separating the symbol mark made by the customer.

[0048] It should be understood that the menu can be scanned in three dimensions by using the structured light method, and a grating is projected to the marked menu, the surface of which is uneven due to the influence of the handwriting pressure, which causes distortion of the grating, and a grating image with the contour information of the menu surface is formed, which is captured by a camera device and saved as a digital image, and after image processing, a picture with only the handwriting pressure trace is obtained. The technology of three-dimensional scanning of the handwriting pressure trace can effectively identify the handwriting made by the customer on the menu and remove the interference caused by the menu background.

[0049] Optionally, the marked menu is subjected to pressure trace feature recognition to determine whether there are numerical and characteristic symbols such as √, ×, and O, if there are numerical and characteristic symbols such as √, ×, and O, the step S102 is continuously executed, and if there are no numerical and characteristic symbols such as √, ×, and O, the step S101 is executed to reacquire the image of the marked menu.

[0050] In another implementation manner of the present application, the marked menu recognition method based on pressure trace detection further comprises: performing character recognition processing on the pressure trace image to obtain a character recognition result, wherein the character recognition result comprises at least one of text, numbers, and special symbols.

[0051] Exemplarily, the character recognition processing on the pressure trace picture is performed by using target detection to recognize text, numbers, and special symbols such as √, ×, and O.

[0052] In another implementation manner of the present application, the frame selection of each pressure trace area in the pressure trace image comprises: using different colors to frame each pressure trace area corresponding to text, numbers, and special symbols in the pressure trace image, respectively.

[0053] Exemplarily, different colors are used to frame each pressure trace area corresponding to text, numbers, and special symbols in the pressure trace image, respectively. For example, red frames can be used to frame special symbols such as √ and O, green frames can be used to frame numbers, and yellow frames can be used to frame text content.

[0054] In another implementation manner of the present application, the position information of each pressure trace area and the text in the menu background image is calculated by using the intersection over union algorithm to obtain the menu name corresponding to each pressure trace area, which comprises: the position information of each pressure trace area and the text in the menu background image is calculated by using the intersection over union algorithm, and the formula of the intersection over union algorithm is as follows:

[0055]

[0056] Wherein, IOU is the intersection over union, B1 and B2 are the areas corresponding to the pressure trace area and the text in the menu background image, respectively; based on the position information, the menu name corresponding to each pressure trace area is determined.

[0057] Exemplarily, the proportion of the framed indentation region in the indentation image and the text frame in the menu background image is calculated, the correct dish is identified, and the dish name position information is output.

[0058] In another implementation of the present application, the dish name and dish quantity corresponding to each indentation region are output, including: determining the dish quantity based on the digital recognition result in the indentation image; transmitting the dish name and dish quantity to the kitchen background; and printing the dish name and dish quantity by the kitchen background.

[0059] Exemplarily, the processor determines the dish quantity based on the digital recognition result in the indentation image, transmits the dish name and dish quantity to the kitchen background, and the kitchen background prints the dish name and dish quantity.

[0060] In another implementation of the present application, the text region in the menu background image is framed in the text detection process, and the text region in the input menu background image can be framed by a PAN module LK-PAN (Large Kernel PAN) with a large receptive field, a FPN structure RSE-FPN (Residual Squeeze-and-Excitation FPN) with a residual attention mechanism, and a DML (Deep Mutual Learning) distillation strategy.

[0061] Among them, LK-PAN is a lightweight PAN (Path Aggregation Network) module with a large receptive field. The main idea is to change the convolution kernel size in the path enhancement of the PAN structure from 3x3 to 9x9, which can improve the receptive field of each pixel of the feature map, making it easier to detect large fonts and extreme aspect ratio texts. RSE-FPN introduces a residual attention mechanism by replacing the convolution layer in FPN with RSEConv to improve the representation ability of the feature map. RSEConv consists of two parts: residual structure and Squeeze-and-Excitation (SE) block, which can effectively improve the text detection performance after being introduced. DML can effectively improve the accuracy of the text detection model by mutual learning between two models with the same structure.

[0062] Four strategies are used in the text frame correction process: light backbone, data augmentation, input resolution, and PACT (Parameterized Clipping Activation) quantization.

[0063] For text recognition, the technology uses a lightweight text recognition network SVTR as the skeleton. Through the SVTR-LCNet lightweight text network, the GTC, the data augmentation strategy TextConAug that mines the text context information, the self-supervised pre-training model TextRotNet, the UDML (Unified-Deep Mutual Learning) and the UIM method, the model is better pre-trained to accelerate the model and improve the effectiveness.

[0064] wherein SVTR-LCNet is a lightweight text recognition network, which combines the transformer-based network SVTR and the CNN-based lightweight network PP-LCNet to achieve lightweight effect. GTC is a strategy trained by attention-guided CTC (Connectionist Temporal Classification). The CTC decoder can achieve faster prediction speed, but the accuracy is low. By introducing the attention mechanism to train CTC, the accuracy can be effectively improved. Text ConAug is a data augmentation strategy that mines text context information. By connecting n different pictures in the batch, the diversity of the data is improved. TextRotNet is a pre-training model obtained by training in a self-supervised manner through rotating different angles with a large amount of unlabeled text line data. We use this pre-training model to initialize the weights of SVTR-LCNet to help the text recognition model converge better.

[0065] It should be understood that UDML is a joint mutual learning strategy that adds Feature loss on the basis of DML. Since the student network and the teacher network used are structurally consistent, only the initial value of the weight is different, when processing the same picture, the same result and feature mapping should be obtained. Feature loss can be used to constrain the distance of the intermediate feature mapping of the two networks, keeping the feature map aligned. By calculating the loss of the feature map output of PP-LCNet, the output of the SVTR module and the output of the Attention module respectively, the accuracy of menu text recognition can be further improved. Using the UDML strategy, the total loss function of the training process includes CTC loss, DML loss and Feature loss, as shown in the following formula:

[0066] loss total =loss ctc +loss dml +loss feat

[0067] Since the student network and the teacher network used by UDML are trained from scratch, CTC loss can be used for network convergence, as shown in the following formula:

[0068] loss ctc = CTC(S hout ,gt) + CTC(T hout ,gt)

[0069] where Shout represents the head output of the student network, Thout represents the head output of the teacher network, and gt gives the groundtruth label of the input image.

[0070] Since it is desired that the final output distributions of the two networks are the same, DML loss is needed to guarantee the consistency of the distribution between the two networks, and the difference between the two sets of probabilities is calculated using KL divergence as loss, as follows:

[0071]

[0072] where KL(p||q) represents the KL divergence of p and q. Spout and Tpout are as follows:

[0073] S pout = Softmax(S hout )

[0074] T pout = Softmax(T hout )

[0075] In the Feature loss, since the structures of the two networks are the same, their feature maps should be the same, and the feature loss can be used to constrain the distance of the intermediate feature maps of the two networks, as follows:

[0076] loss feat = L2(S bout , T bout )

[0077] where Sbout represents the backbone output of the student network, and Tbout represents the backbone output of the teacher network. Here, the mean square error loss is used.

[0078] UIM is a simple unsupervised data mining strategy. The main idea is to use a high-precision text recognition model to predict unlabeled images to obtain pseudo labels, and select samples with high prediction confidence as training data to train a lightweight model.

[0079] The super-lightweight OCR system PP-OCRv3 technology is adopted. Compared with traditional OCR technology, the effectiveness of this technology is greatly improved, and it can more accurately identify font features.

[0080] According to a second aspect of the present application, a marked menu recognition device 300 based on indentation detection is provided, characterized in that it comprises:

[0081] An image acquisition module 301 is configured to acquire a marked menu image.

[0082] An image processing module 302 is configured to perform image segmentation processing based on the indentation in the marked menu image to obtain an indentation image and a menu background image; frame each indentation region in the indentation image; detect and recognize the text in the indentation region and the menu background image based on a text detection and recognition model to obtain the meaning corresponding to each indentation and each dish name in the menu background image, wherein the meaning corresponding to the indentation includes at least one of a text meaning, a number meaning, and a special symbol meaning; and calculate the position information of the text in each indentation region and the menu background image by an intersection over union algorithm to obtain the dish name corresponding to each indentation region.

[0083] A result output module 303 is configured to output the dish name corresponding to each indentation region and the number of dishes.

[0084] In the marked menu recognition device 300 based on indentation detection of the present application, the image of the indentation region in the marked menu image and the menu background image are extracted separately, the character meaning analysis is performed on the indentation image, the obtained character meaning is more accurate, the menu background image is extracted from the marked menu image, the text recognition is performed on the extracted menu background image to obtain each dish name, the indentation region and the dish name can be well corresponded in the subsequent process, the position information of the text in each indentation region and the menu background image is calculated by the intersection over union algorithm, the dish name corresponding to the indentation region is more accurate, the working efficiency is ensured, and the accuracy of the menu marking content recognition is improved.

[0085] In another implementation manner of the present application, the image acquisition module 301 is further configured to monitor whether a menu is placed under the camera device through a sensor; if it is monitored that the menu is placed under the camera device, the camera device and the light supplement device automatically start working; and the marked menu image is acquired based on the camera device and the light supplement device.

[0086] In another implementation manner of the present application, the image processing module 302 is further configured to perform recognition processing on the marked menu image based on a three-dimensional scanning indentation recognition technology to obtain an indentation recognition result; and perform image segmentation processing according to the indentation recognition result to obtain the indentation image and the menu background image.

[0087] In another implementation manner of the present application, the image processing module 302 is further configured to perform character recognition processing on the indentation image to obtain a character recognition result, wherein the character recognition result includes at least one of a text, a number, and a special symbol.

[0088] In another implementation manner of the present application, the image processing module 302 is further configured to frame the indentation area corresponding to each text, number and special symbol in the indentation image respectively using different colors.

[0089] In another implementation manner of the present application, the image processing module 302 is further configured to calculate the position information of each indentation area and the text in the menu background image by using the intersection over union algorithm, and the formula of the intersection over union algorithm is as follows:

[0090]

[0091] wherein, IOU is the intersection over union, B1 and B2 are the areas corresponding to the indentation area and the text in the menu background image respectively; based on the position information, the dish name corresponding to each indentation area is determined.

[0092] In another implementation manner of the present application, the result output module 303 is further configured to determine the dish quantity based on the recognition result of the number in the indentation image; transmit the dish name and the dish quantity to the kitchen background; and the kitchen background prints the dish name and the dish quantity.

[0093] As shown in Figure 4 The electronic device 400 can include a processor 401, a memory 403, and a communication bus 404, a communication interface 405.

[0094] Among them:

[0095] The processor 401, the memory 403 and the communication interface 405 complete the communication among each other through the communication bus 404.

[0096] The communication interface 405 is configured to communicate with other electronic devices or servers.

[0097] The processor 401 is configured to execute the program 402, and specifically can execute the steps of any one of the above-mentioned implementation manners of the marked menu recognition method based on indentation detection.

[0098] Specifically, the program 402 can include program code, and the program code includes computer operation instructions.

[0099] The processor 401 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the application. The one or more processors included in the smart device can be of the same type, such as one or more CPUs, or can be of different types, such as one or more CPUs and one or more ASICs.

[0100] The memory 403 is configured to store the program 402. The memory 403 can include a high-speed RAM memory, and can further include a non-volatile memory, such as at least one disk memory.

[0101] The program 402 can be specifically configured to cause the processor 401 to perform the steps of any of the marked menu recognition methods based on indentation detection described in the embodiments. The specific implementation of each step in the program 402 can refer to the corresponding description of the steps and units performed by any of the above-described marked menu recognition methods based on indentation detection, which will not be described here. It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the foregoing method embodiments.

[0102] The exemplary embodiments of the present application further provide a non-transitory computer-readable storage medium having computer instructions stored therein, wherein the computer instructions are used to cause a computer to execute the method of any of the embodiments of the present application.

[0103] The above-described method according to the embodiments of the present application can be implemented in hardware, firmware, or as software or computer code stored in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk, or an optical disk, or downloaded from a network and stored in a local recording medium, so that the method described herein can be processed by such software using a general purpose computer, a special purpose processor, or programmable or special purpose hardware (such as an ASIC or an FPGA). It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component (for example, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor, or hardware, the method described herein is implemented. In addition, when a general purpose computer accesses the code for implementing the method shown herein, the execution of the code will convert the general purpose computer into a special purpose computer for executing the method shown herein.

[0104] To this end, particular embodiments of the application have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0105] It should be noted that all directional indications, such as upper, lower, left, right, front, back, rear, etc., are merely used for convenience of description and are not intended to limit the application to a particular orientation.

[0106] In the description of the present application, the terms "first", "second", etc., are used only for convenience and are not intended to imply or suggest a relative importance or a particular order of the technical features. Therefore, the features with "first", "second" can explicitly or implicitly include at least one of the features.

[0107] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0108] It should be noted that, although the specific embodiments of the present application are described in detail in conjunction with the accompanying drawings, it should not be understood as a limitation on the scope of protection of the present application. Various modifications and variations of the embodiments described in the claims are still within the scope of protection of the present application without creative labor.

[0109] The examples of the embodiments of the present application are intended to simply illustrate the technical features of the embodiments of the present application, so that those skilled in the art can intuitively understand the technical features of the embodiments of the present application, and are not improper limitations on the embodiments of the present application.

[0110] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A marked menu recognition method based on indentation detection, characterized by, The method comprises the following steps: acquiring a marked menu image; performing image segmentation processing based on indentations in the marked menu image to obtain an indentation image and a menu background image, which comprises the following steps: performing identification processing on the marked menu image based on an indentation identification technology of three-dimensional scanning to obtain an indentation identification result; performing image segmentation processing according to the indentation identification result to obtain an indentation image and a menu background image; framing each indentation region in the indentation image; detecting and identifying texts in the indentation region and the menu background image based on a text detection and identification model to obtain meanings corresponding to each indentation and each dish name in the menu background image, wherein the meanings corresponding to each indentation comprise at least one of a text meaning, a number meaning and a special symbol meaning; calculating position information of texts in each indentation region and the menu background image by an intersection over union algorithm to obtain dish names corresponding to each indentation region; outputting dish names corresponding to each indentation region and dish quantities.

2. The method of claim 1, wherein, The method further comprises the following steps: monitoring whether a menu is placed under a camera device by a sensor; if it is monitored that a menu is placed under the camera device, the camera device and a light supplement device automatically start working. The method further comprises the following steps: acquiring a marked menu image based on the camera device and the light supplement device.

3. The method of claim 1, wherein, The method further comprises the following steps: performing character recognition processing on the indentation image to obtain a character recognition result, wherein the character recognition result comprises at least one of a text, a number and a special symbol.

4. The method of claim 3, wherein, The method further comprises the following steps: framing each indentation region in the indentation image by using different colors respectively.

5. The method of claim 4, wherein, The method further comprises the following steps: calculating position information of texts in each indentation region and the menu background image by an intersection over union algorithm, and the formula of the intersection over union algorithm is as follows: wherein, IOU is an intersection over union, B1 and B2 are regions corresponding to the indentation region and the texts in the menu background image respectively; determining dish names corresponding to each indentation region based on the position information.

6. The method of claim 3, wherein, The method further comprises the following steps: determining the dish quantities based on number recognition results in the indentation image; transmitting the dish names and the dish quantities to a kitchen back-end; printing the dish names and the dish quantities by the kitchen back-end.

7. A marked menu recognition apparatus based on indentation detection, characterized by, The method comprises the following steps: an image acquisition module, which is configured to acquire a marked menu image; An image processing module: based on the indentation in the marked menu image, image segmentation processing is performed to obtain an indentation image and a menu background image, including: based on the indentation recognition technology of three-dimensional scanning, the marked menu image is recognized to obtain an indentation recognition result; according to the indentation recognition result, image segmentation processing is performed to obtain an indentation image and a menu background image; each indentation region in the indentation image is framed; based on a text detection and recognition model, the text in the indentation region and the menu background image is detected and recognized to obtain the meaning corresponding to each indentation and each dish name in the menu background image, wherein the meaning corresponding to the indentation includes at least one of a text meaning, a digital meaning and a special symbol meaning; through a intersection over union algorithm, the position information of the text in each indentation region and the menu background image is calculated to obtain the dish name corresponding to each indentation region; A result output module: used for outputting the dish name corresponding to each indentation region and the dish quantity.

8. An electronic device, comprising: Including: A memory, a processor and a computer program stored on the memory and executable on the processor, when the processor executes the computer program, the steps of the marked menu recognition method based on indentation detection according to any one of claims 1 to 6 are implemented.

9. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the processor executes the computer program, the steps in the marked menu recognition method based on indentation detection according to any one of claims 1 to 6 are implemented.

Images