Intelligent dish matching method

Abstract

The invention discloses an intelligent dish matching method, which comprises the following operation steps: acquiring image data at the current moment, cutting an image, retaining a dining area image, cutting a table area image from the dining area image, zooming the table area image to 1280 * 1280, sending the table area image into a detection network, and obtaining a detection result, namely the position information of a dinner plate. Mapping the position information of the dinner plate to original coordinates, sending the table area image to a feature extraction network to obtain four feature points, calculating a perspective transformation matrix according to the coordinates of the feature points and the alignment coordinates, calculating the table area image after perspective transformation according to the obtained perspective transformation matrix, and calculating the table area image after perspective transformation according to the obtained perspective transformation matrix. And calculating perspective transformation coordinates of the dinner plate position mapping coordinates. According to the method, dish detection is replaced by dinner plate detection, more similar features exist among dinner plates, the detection accuracy can be improved, and repeated network training can be avoided through a database checking mode.

Description

Technical Field

[0001] The present invention relates to the field of intelligent catering technology, and in particular to an intelligent dish pairing method. Background Art

[0002] The traditional catering industry has entered the era of smart catering. The essence of smart catering is to use smart hardware and software, Internet technology, big data, etc. to drive the reconstruction of people, goods, and venues in the catering industry, improve consumers' user experience and the service capabilities of catering companies. Smart catering has functions such as smart reservation, queuing, queue jumping, scanning code ordering, and smart dish matching;

[0003] Smart dish pairing refers to the process of using artificial intelligence technology to identify, classify and evaluate dishes, eliminating the need for manual dish pairing and improving efficiency. Compared with traditional dish pairing methods, smart dish pairing has the advantages of automation, accuracy, quick feedback, data analysis and improved user experience. Smart dish pairing can improve the operational efficiency of catering companies.

[0004] However, in the actual use of the existing technology, the implementation process of the general intelligent dish matching is mainly based on network testing, and the location of the specified dish is detected through network testing;

[0005] The prior art has the following problems:

[0006] 1. The accuracy is not high. One reason is that sometimes the images are blurry and the features are not obvious. Another reason is that some dishes are highly similar.

[0007] 2. New dishes require retraining of the network. The output dimension of the parameter file obtained after training is generally fixed to the dimension during training. New dishes that do not belong to the original classification need to be added to the data set and the number of classifications needs to be modified.

[0008] 3. Each modification cycle is long and multiple areas need to be modified. Summary of the invention

[0009] The purpose of the present invention is to provide an intelligent dish pairing method to solve the low accuracy rate in the implementation process of the common intelligent dish pairing proposed in the above background technology. One reason is that the image is sometimes blurred and the features are not obvious, and the other is that some dishes have a high similarity; the network needs to be retrained for new dishes. The parameter file obtained after training generally has an output dimension fixed to the dimension during training. The newly added dishes do not belong to the original classification and need to add new data to the data set and modify the number of classifications; each modification cycle is long, and multiple areas need to be modified.

[0010] To achieve the above object, the present invention provides an intelligent dish pairing method, comprising the following steps:

[0011] S1: Get the image data at the current moment;

[0012] S2: crop the image and keep the dining area image;

[0013] S3: cropping the table area image from the dining area image;

[0014] S4: Scale the table area image to 1280*1280 and send it to the detection network to obtain the detection result of the plate position information;

[0015] S5: Mapping the position information of the dinner plate back to the original coordinates;

[0016] S6: Send the table area image to the feature extraction network to obtain four feature points (the four corner points of the table);

[0017] S7: Calculate the perspective transformation matrix according to the feature point coordinates and the alignment coordinates;

[0018] S8: Calculate the table area image after perspective transformation according to the obtained perspective transformation matrix;

[0019] S9: Calculate the perspective transformation coordinates of the dinner plate position mapping coordinates according to the obtained perspective transformation matrix;

[0020] S10: According to the table area image and the perspective transformation coordinates, the plate area image is cropped to obtain a set of plate area thumbnails after perspective transformation;

[0021] S11: extracting feature points of each plate image in the plate area thumbnail set after perspective transformation to obtain four plate feature points (plate corner points), calculating an affine transformation matrix according to the feature point coordinates and alignment coordinates, and obtaining a set of plate images after affine transformation according to the affine transformation matrix;

[0022] S12: Send each image in the set of dinner plate images after affine transformation to a feature extraction network to obtain a set of feature vectors;

[0023] S13: Each feature in the feature vector set is compared with the feature vector in the feature library, and the one with the highest matching degree is the dish classification corresponding to the plate image.

[0024] Preferably, the current image data is I t The dining area image is Roi t The table area image is Table t .

[0025] Preferably, the plate position information is The plate position information is mapped back to the original coordinates to obtain

[0026] Preferably, the four feature points are Corner t ={(cx1, cy1), (cx2, cy2), (cx3, cy3), (cx4, cy4)}.

[0027] Preferably, the perspective transformation matrix is ​​A t =Perspective(Corner t , Align).

[0028] Preferably, the table area image is a PerspectiveTable t , the perspective transformation coordinates are The set of small images of the plate area after perspective transformation is Dishes t .

[0029] Preferably, the set of dinner plate images after affine transformation is Dishes′ t , the feature vector set

[0030] Preferably, the feature library is composed of feature vectors extracted from dishes through a feature extraction network. Each dish corresponds to a feature vector. When adding a new dish, the dish features can be extracted and entered into the library. When checking the dish, the features are also obtained through feature extraction and compared in the feature library.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The present invention uses plate detection instead of dish detection. Plates have more similar features, so the detection accuracy will be improved. The database query method can avoid repetitive network training;

[0033] 2. The present invention also does not need to adjust the code according to different dishes, and only needs to add a dish library. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the process flow of an intelligent dish pairing method of the present invention;. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] See also Figure 1The present invention provides an intelligent dish pairing method, comprising the following steps:

[0037] S1: Get the current image data I t ;

[0038] S2: Crop the image and keep the dining area image ROI t ;

[0039] S3: Crop the table area image from the dining area image t ;

[0040] S4: Scale the table area image to 1280*1280 and send it to the detection network to obtain the detection result of the plate position information

[0041] S5: Map the plate position information back to the original coordinates to obtain

[0042] S6: Table area image Table t Send it to the feature extraction network to get four feature points (the four corners of the table) Corner t ={(cx1, cy1), (cx2, cy2), (cx3, cy3), (cx4cy4)};

[0043] S7: Calculate the perspective transformation matrix based on the feature point coordinates and alignment coordinates

[0044] A t =Perspective(Corner t , Align);

[0045] S8: According to the obtained perspective transformation matrix A t Calculate the perspective transformed table area image

[0046] PerspectiveTable t ;

[0047] S9: According to the obtained perspective transformation matrix, the perspective transformation coordinates of the plate position mapping coordinates are calculated to obtain

[0048] S10: According to PerspectiveTable t as well as Crop the plate area image to get the perspective transformed plate area image set Dishes t ;

[0049] S11: Dishes collection tExtract feature points from each plate image in , and obtain four plate feature points (plate corner points). Calculate the affine transformation matrix according to the feature point coordinates and alignment coordinates, and calculate the affine transformation plate image set Dishes′ according to the affine transformation matrix t ;

[0050] S12: Set the plate image set Dishes′ t Each image is sent to the feature extraction network to obtain a set of feature vectors

[0051] S13: Feature vector set Each feature in is compared with the feature vector in the feature library, and the one with the highest matching degree is the dish classification corresponding to the plate image;

[0052] The feature library is composed of feature vectors extracted from dishes through the feature extraction network. Each dish corresponds to a feature vector. When adding a new dish, you only need to extract the dish features and enter them into the library. When checking dishes, you can also extract features and compare them in the feature library.

[0053] When the invention is in use, the detection of plates is used instead of the detection of dishes, wherein the plates have more similar features, so that the detection accuracy is improved. At the same time, repetitive network training can be avoided by checking the library. For this solution, there is no need to adjust the code according to different dishes, only to add the dish library.

[0054] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0055] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent dish pairing method, characterized by: The steps are as follows: S1: Get the image data at the current moment; S2: crop the image and keep the dining area image; S3: cropping the table area image from the dining area image; S4: Scale the table area image to 1280*1280 and send it to the detection network to obtain the detection result of the plate position information; S5: Mapping the position information of the dinner plate back to the original coordinates; S6: Send the table area image to the feature extraction network to obtain four feature points (the four corner points of the table); S7: Calculate the perspective transformation matrix according to the feature point coordinates and the alignment coordinates; S8: Calculate the table area image after perspective transformation according to the obtained perspective transformation matrix; S9: Calculate the perspective transformation coordinates of the dinner plate position mapping coordinates according to the obtained perspective transformation matrix; S10: According to the table area image and the perspective transformation coordinates, the plate area image is cropped to obtain a set of plate area thumbnails after perspective transformation; S11: extracting feature points of each plate image in the plate area thumbnail set after perspective transformation to obtain four plate feature points (plate corner points), calculating an affine transformation matrix according to the feature point coordinates and alignment coordinates, and obtaining a set of plate images after affine transformation according to the affine transformation matrix; S12: Send each image in the set of dinner plate images after affine transformation to a feature extraction network to obtain a set of feature vectors; S13: Each feature in the feature vector set is compared with the feature vector in the feature library, and the one with the highest matching degree is the dish classification corresponding to the plate image.

2. The intelligent dish pairing method according to claim 1, characterized in that: The current moment image data is I t , the dining area image is Roi t , the table area image is Table t .

3. The intelligent dish pairing method according to claim 2, characterized in that: The plate position information is The plate position information is mapped back to the original coordinates to obtain 4. The intelligent dish pairing method according to claim 3, characterized in that: The four feature points are Corner t ={(cx1, cy1), (cx2, cy2), (cx3, cy3), (cx4, cy4)}.

5. The intelligent dish pairing method according to claim 4, characterized in that: The perspective transformation matrix is ​​A t =Perspective(Corner t , Align).

6. The intelligent dish pairing method according to claim 5, characterized in that: The table area image is a PerspectiveTable t , the perspective transformation coordinates are The set of small images of the plate area after perspective transformation is Dishes t .

7. The intelligent dish pairing method according to claim 6, characterized in that: The set of dinner plate images after the affine transformation is Dishes′ t , the feature vector set 8. The intelligent dish pairing method according to claim 7, characterized in that: The feature library is composed of feature vectors extracted from dishes through a feature extraction network. Each dish corresponds to a feature vector. When adding a new dish, the dish features can be extracted and entered into the library. When checking a dish, the features are also obtained through feature extraction and compared in the feature library.

Images