Method and apparatus for beverage cabinet image recognition, device

Abstract

The application relates to the technical field of intelligent devices, and discloses a method and device for beverage cabinet image recognition and equipment. The method comprises the following steps: acquiring at least two beverage images of a beverage cabinet; performing image detection on each beverage image to detect beverage cabinet layer position information and the size information of each layer of beverage in each beverage image; performing splicing processing on each beverage image according to the beverage cabinet layer position information and the size information of each layer of beverage to obtain a to-be-recognized image; and recognizing beverage information in the to-be-recognized image based on a deep learning algorithm. In this way, the automatic acquisition of beverage information in the beverage cabinet is realized, and the intelligence of the beverage cabinet is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent devices, for example to a method and device for beverage cabinet image recognition. BACKGROUND

[0002] A beverage cabinet is a refrigerator specially used for refrigerating beverages. The refrigerator can refrigerate a single kind of beverage or multiple kinds of beverages. When the quantity of a certain kind of beverage in the beverage cabinet decreases, it may need to be replenished in time. Therefore, it is necessary to check the kind and corresponding quantity of the beverage in the beverage cabinet in time, that is, to obtain the beverage information in the beverage cabinet in time.

[0003] At present, the beverage information in the beverage cabinet is generally obtained by manual inspection, which greatly consumes time and human resources. SUMMARY

[0004] The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. The summary is not an extensive overview of the application. It is not intended to identify key / critical elements of the embodiments or to delineate the scope of the embodiments. Its sole purpose is to present some embodiments in a simplified form as a prelude to the more detailed description that is presented later.

[0005] The embodiments of the present disclosure provide a method, device and equipment for beverage cabinet image recognition to solve the technical problem of low intelligence of the beverage cabinet.

[0006] In some embodiments, the method comprises:

[0007] obtaining at least two beverage images of a beverage cabinet;

[0008] performing image detection on each of the beverage images to detect beverage cabinet shelf position information and size information of each layer of beverage in each of the beverage images;

[0009] performing splicing processing on each of the beverage images according to the beverage cabinet shelf position information and the size information of each layer of beverage to obtain a to-be-recognized pattern;

[0010] recognizing beverage information in the to-be-recognized pattern based on a deep learning algorithm.

[0011] In some embodiments, the device comprises:

[0012] an obtaining module configured to obtain at least two beverage images of a beverage cabinet;

[0013] a detection module configured to perform image detection on each of the beverage images to detect beverage cabinet shelf position information and size information of each layer of beverage in each of the beverage images;

[0014] The splicing module is configured to splice each of the beverage images according to the beverage cabinet layer position information and the size information of each layer of beverages to obtain a to-be-recognized pattern.

[0015] The recognition module is configured to recognize beverage information in the to-be-recognized pattern based on a deep learning algorithm.

[0016] In some embodiments, the device for beverage cabinet image recognition includes a processor and a memory storing program instructions, and the processor is configured to execute the program instructions to perform the method for beverage cabinet image recognition.

[0017] In some embodiments, the device includes the device for beverage cabinet image recognition.

[0018] The method, device and equipment for beverage cabinet image recognition provided by the embodiments of the present disclosure can achieve the following technical effects:

[0019] The beverage cabinet image detection can be performed on each of the obtained beverage images to detect beverage cabinet layer position information and size information of each layer of beverages in each of the beverage images, and then each of the beverage images is spliced according to the beverage cabinet layer position information and the size information of each layer of beverages to obtain a to-be-recognized pattern, and then the deep learning algorithm is used to recognize beverage information of the beverage cabinet. In this way, the automatic acquisition of beverage information in the beverage cabinet is realized, and the intelligence of the beverage cabinet is improved. Moreover, the image splicing is performed according to the beverage cabinet layer position information and the size information of each layer of beverages detected by the image detection, which is not affected by factors such as light, shooting position, consistency of collection equipment and image distortion, so that the image splicing of the beverage cabinet is complete, and the accuracy of beverage detection and recognition is improved.

[0020] The general description above and the following description below are exemplary and explanatory only and are not intended to be limiting. BRIEF DESCRIPTION OF DRAWINGS

[0021] One or more embodiments are illustrated by way of example in the figures that are not intended to be limiting of the embodiments as disclosed herein. Like numbers refer to like elements throughout the description, some of which numbers have been intentionally left off in order to improve clarity of the description. The figures are not necessarily drawn to scale, and that, in some instances, various drawings maybe omitted or simplified in order not to obscure one or more aspects of the embodiments. The drawings are intended to facilitate understanding of examples of embodiments, but in no way limit the scope of what is described herein.

[0022] Figure 1 is a flowchart of a method for beverage cabinet image recognition provided by an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of a beverage image after image detection provided by an embodiment of the present disclosure;

[0024] Figure 3is a flowchart of a process for obtaining a beverage cabinet image provided by an embodiment of the present disclosure.

[0025] Figure 4 is a flowchart of a beverage cabinet image recognition method provided by an embodiment of the present disclosure.

[0026] Figure 5 is a structural diagram of a beverage cabinet image recognition device provided by an embodiment of the present disclosure.

[0027] Figure 6 is a structural diagram of a beverage cabinet image recognition device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to enable a more detailed understanding of the features and technical content of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below in conjunction with the drawings, which are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to facilitate the drawings.

[0029] The terms "first", "second", and the like in the specification and claims of the embodiments of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0030] Unless otherwise specified, the term "a plurality of" means two or more.

[0031] In the embodiments of the present disclosure, the character " / " represents an "or" relationship between the objects before and after it. For example, A / B represents: A or B.

[0032] The term "and / or" is a description of the relationship between the objects, which means that there can be three relationships. For example, A and / or B, which means: A or B, or, A and B, three relationships.

[0033] The beverage cabinet is a kind of refrigerator specially used for refrigerating beverages. In the embodiment of the present disclosure, image detection can be performed on each beverage image of the beverage cabinet obtained, and stitching processing is performed to obtain a to-be-recognized pattern, and then beverage information of the beverage cabinet is recognized based on a deep learning algorithm. In this way, automatic acquisition of beverage information in the beverage cabinet is realized, and the intelligence of the beverage cabinet is improved. Moreover, through the beverage cabinet layer position information and the size information of each layer of beverages detected by image detection, image stitching is performed, which is not affected by factors such as light, shooting position, consistency of collection equipment and image distortion, so that the completeness of the image stitching of the beverage cabinet is good, and the accuracy of beverage detection and recognition is further improved.

[0034] Figure 1 is a flowchart of a beverage cabinet image recognition method provided by the embodiment of the present disclosure. As shown in Figure 1 , the process for beverage cabinet image recognition includes:

[0035] Step 101: Obtain at least two beverage images of the beverage cabinet.

[0036] Generally, two or more image collection devices can be configured in the beverage cabinet, for example, one camera is configured at the top center and the bottom center of the beverage cabinet, or one camera is configured at the top, middle and bottom of the frame of the beverage cabinet. Thus, at least two beverage images of the beverage cabinet can be obtained through the configured image collection devices.

[0037] The image collection device can collect images of the beverage cabinet in real time, or collect images of the beverage cabinet only when the set conditions are met. For example, generally, the information of the refrigerated beverage can change only when the beverage cabinet door is opened, therefore, in some embodiments, the image collection device is started to collect the beverage images in the beverage cabinet only when it is determined that the beverage cabinet door is opened.

[0038] Of course, the beverage cabinet can locally obtain at least two beverage images of the beverage cabinet collected by the image collection device, and perform image processing and recognition, or the beverage cabinet sends at least two beverage images of the beverage cabinet collected by the image collection device to the server in the cloud, so that the server obtains at least two beverage images sent by the beverage cabinet and performs image processing and recognition.

[0039] Among them, the beverage cabinet locally obtains images for corresponding processing and recognition, which can improve the efficiency of beverage cabinet image recognition, while the server in the cloud obtains images from the beverage cabinet and performs corresponding processing and recognition, which can reduce the memory occupation of the beverage cabinet, improve the running speed of the beverage cabinet, and the server uniformly manages the beverage information of each beverage cabinet, further improving the dataization and intelligence of the beverage cabinet.

[0040] Step 102: image detection is performed on each beverage image to detect the beverage cabinet shelf position information and the size information of each layer of beverage in each beverage image.

[0041] Image detection has always been a popular application field in pattern recognition machine learning. Common image detection includes image segmentation, image classification, target detection, etc. Among them, image segmentation is a technology and process of finding the area where the target is located from the image, dividing the image into several specific areas with unique properties, and proposing the target of interest. Common image segmentation algorithms include threshold-based segmentation algorithm, clustering-based segmentation algorithm, edge-based segmentation algorithm, region growing-based segmentation algorithm, and graph-based segmentation algorithm. These algorithms perform differently in terms of segmentation efficiency and accuracy.

[0042] Image classification is to determine the category of objects in an image. For example, cat and dog classification, to determine whether the image is a cat image or a dog image. For image classification, the most important thing is to find suitable features, and then through these features, using suitable classification algorithms, the image can be easily classified. Common features include gray level histogram-based features, morphological-based features, and texture-based features, LBP, SIFT features, and excellent classification algorithms such as random forest, AdBoost, and SVM.

[0043] Target detection, also known as target extraction, is an image segmentation based on target geometry and statistical features, which combines target segmentation and recognition, locates the target, and determines the target position and size.

[0044] At present, with the great development of deep learning in the field of image, more and more application scenarios use deep learning algorithm, which not only makes the image detection more accurate, but also greatly improves the real-time performance.

[0045] In some embodiments, the image detection based on deep learning algorithm can include image detection based on fasterRcnn algorithm. After normalizing the image, input the VGG16 convolutional neural network for processing to obtain a feature image, then generate a detection frame of the feature image through RPN, and detect the target in the image according to the detection frame to obtain the corresponding target information.

[0046] In step 101, two or more beverage images of the beverage cabinet are obtained, and the target in the beverage image is the beverage and the shelf where the beverage is placed. Therefore, through image detection based on deep learning algorithm, the beverage cabinet shelf position information and the size information of each layer of beverage in each beverage image can be detected.

[0047] Since the acquired beverage images can have some image distortion, in some embodiments, image preprocessing including image distortion correction can be performed on each beverage image before image detection based on a deep learning algorithm, that is, image detection on each beverage image includes: performing image preprocessing including image distortion correction on each beverage image to obtain a corresponding preprocessed image; based on a deep learning target detection algorithm, detecting each preprocessed image to obtain beverage cabinet layer position information and each layer of beverage size information in each preprocessed image.

[0048] Among them, the image preprocessing includes image distortion correction. For example: the formula from the world coordinate to the camera coordinate is as follows:

[0049]

[0050] Camera coordinate system to image coordinate system:

[0051]

[0052] Where A is the intrinsic matrix, which is determined by the camera internal hardware

[0053]

[0054] Normalization

[0055]

[0056] u=f x .x″+c x

[0057] v=f y .y″+c y

[0058] Where

[0059]

[0060] Where

[0061] r 2 =x′+y′

[0062] Parameters: is the radial distortion coefficient, 2p2x′y′+p1(r 2 +2y′ 2 ) is the tangential distortion coefficient.

[0063] It should be noted that k1-k6 are radial distortion parameters, p1 and p2 are tangential distortion parameters, s1 and s2 are thin prism distortion parameters of the image acquisition device, which are generally ignored, x' is an ideal image pixel coordinate or point coordinate without distortion, and x" is an image pixel or point coordinate with lens distortion parameters. Generally, after obtaining the image, x' and y" are usually known, that is, the image with distortion is obtained through the image acquisition device. Therefore, it is necessary to calculate back to obtain x' and y', and the distortion map point is obtained through the correction image, that is, x" and y" are obtained through x' and y'.

[0064] Therefore, in the embodiments of the present disclosure, two or more beverage images are obtained, and image preprocessing including image distortion correction needs to be performed on each image, which can include: determining the distortion coordinates (x", y") of the set image pixels in the current beverage image; determining the distortion correction coordinates (x', y') corresponding to the set image pixels and the distortion coordinates in the current beverage image through formula (1); and generating a corresponding current preprocessing image according to each distortion correction coordinate (x', y').

[0065]

[0066] wherein r 2 = x' + y', is a radial distortion coefficient, 2p2x'y' + p1(r 2 + 2y' 2 ) is a tangential distortion coefficient; k1-k6 are radial distortion parameters, p1 and p2 are tangential distortion parameters, and s1 and s2 are thin prism distortion parameters of the image acquisition device.

[0067] After the image preprocessing including image distortion correction is performed on each beverage image to obtain the corresponding preprocessing image, image detection based on a deep learning algorithm can be used to detect the beverage cabinet shelf position information and the size information of each layer of beverage in each preprocessing image. In some embodiments, image detection based on the faster Rcnn algorithm can be used, which can include: inputting the current preprocessing image after normalization into a VGG16 convolutional neural network for processing to obtain a current feature image; obtaining a detection frame of the current feature image according to the Faster RCNN algorithm, and obtaining the beverage cabinet shelf position information and the size information of each layer of beverage in the current preprocessing image according to the detection frame.

[0068] Step 103: performing splicing processing on each beverage image according to the beverage cabinet shelf position information and the size information of each layer of beverage to obtain a to-be-recognized pattern.

[0069] The beverage cabinet shelf position information and the size information of each layer of beverage in each beverage image have been detected, and thus the first height value HH between the lowest shelf and the lower edge of the corresponding beverage image in each beverage image, the second height h of the lowest layer of beverage, and the third height value of the beverage in other shelves can be determined. Generally, the specifications of beverages are known, and thus each type of beverage has a corresponding preset beverage height H.

[0070] When two or more beverage images are acquired by the image acquisition device, the lowermost beverage of the previous image generally overlaps with the uppermost beverage of the next image due to the overlapping of the shooting range of the image acquisition device. Therefore, the acquired beverage images need to be sorted, cut, spliced, and the like to form a to-be-recognized image including the beverages in the complete beverage cabinet. In the related art, the acquired images can be spliced according to image parameters such as shadow, brightness, and the like to obtain the to-be-recognized image. In some embodiments of the present disclosure, the acquired beverage images can also be spliced according to the beverage cabinet shelf position information and the size information of each layer of beverage to obtain the to-be-recognized image.

[0071] In some embodiments, the acquired beverage images can be sorted by the order of the image acquisition device, or the sorting of the acquired beverage images can include: determining first beverage information of each beverage image in which the height of the beverage is less than the preset beverage height H according to the size information of each layer of beverage, and then splicing and sorting each beverage image according to the first beverage information.

[0072] After splicing and sorting each beverage image, the image can be cropped. In some embodiments, the cropping of the image can include: determining the first height value HH between the current lowest shelf and the lower edge of the current beverage image according to the beverage cabinet shelf position information in the current beverage image; determining the second height h of the lowest layer of beverage according to the size information of each layer of beverage in the current beverage image; and in a case where the first height value HH is less than the preset beverage height H or the second height h is less than the preset beverage height H, determining the lower edge of the current beverage image as the current lowest shelf and performing the cropping.

[0073] For example, two or more beverage images of a beverage cabinet with one layer of redundant beverage are acquired, each beverage image is corrected, and each image is detected and recognized by a deep learning detection algorithm to obtain four possible cases of each beverage image as shown in Figure 2 .

[0074] Figure 2 is a beverage image schematic diagram after image detection provided by an embodiment of the present disclosure. As shown in Figure 2As shown, the preset beverage height is H, the height between the lowest shelf and the lower edge of the beverage image is the first height value HH, and the second height of the lowest beverage is h. If HH < or h < H, the lower edge of the beverage image is cut off along HH. Figure 2 In the upper left corner of the middle, HH > H and h is also not less than H, so no cutting is needed, and in the other three cases, h < H, so the lower edge of the beverage image needs to be determined as the minimum shelf, that is, the cutting process is performed from the minimum shelf of the beverage image.

[0075] Because there is one layer of beverage redundancy in the beverage image, according to the splicing order, the next beverage image must contain one layer of cut-off beverage. In this way, the next beverage image is also sequentially cut off, and the cut-off image is obtained.

[0076] Finally, the cut-off beverage image is spliced to obtain the complete beverage image in the beverage cabinet, that is, the to-be-recognized pattern.

[0077] Step 104: Based on the deep learning algorithm, the beverage information in the to-be-recognized pattern is recognized.

[0078] After obtaining the complete beverage image in the beverage cabinet, that is, the to-be-recognized pattern, the recognition processing based on the deep learning algorithm is performed, and the beverage information in the beverage cabinet is obtained.

[0079] The algorithm model of the deep learning algorithm is configured and saved after machine learning based on the obtained sample, and therefore, in some embodiments, before the beverage information in the to-be-recognized pattern is recognized based on the deep learning algorithm, the method further includes: obtaining sample images of at least two size types of the same beverage, wherein the height of the beverage in one size type is less than the preset beverage height H; and based on a ResNet model, training the sample images to obtain a ResNet deep learning algorithm model. In this embodiment, the obtained sample images have multiple size specifications, for example, sample images with a complete beverage height H, sample images with 1 / 2 beverage height, sample images with 1 / 3 beverage height, and the like. In this way, because of the diversity of the sample, the beverage information of beverages of various heights in the to-be-recognized pattern can also be recognized, further improving the accuracy of image recognition.

[0080] After the ResNet deep learning algorithm model is configured, the to-be-recognized pattern is input into the ResNet deep learning algorithm model, and the beverage information of each beverage in the beverage cabinet can include information such as the type and quantity of the beverage.

[0081] Of course, the embodiments of the present disclosure are not limited to the ResNet deep learning algorithm model, and other image recognition algorithms based on a convolutional neural network CNN can also be applied, which will not be enumerated one by one.

[0082] It can be seen that in the embodiment, image detection can be performed on each beverage image of the beverage cabinet obtained, and stitching processing is performed to obtain a to-be-recognized pattern, and then beverage information of the beverage cabinet is recognized based on a deep learning algorithm. In this way, automatic acquisition of beverage information in the beverage cabinet is realized, and the intelligence of the beverage cabinet is improved. Moreover, the beverage cabinet layer position information and the size information of each layer of beverages detected through image detection are used for image stitching, which is not affected by factors such as light, shooting position, consistency of collection equipment, and image distortion, so that the image stitching completeness of the beverage cabinet is good, and the accuracy of beverage detection and recognition is improved. In addition, the sample pictures used in the deep learning algorithm model include sample pictures of various sizes, complete bottle heights, 1 / 2 beverage heights, 1 / 3 beverage heights, and the like. In this way, due to the diversity of the samples, the beverage information of beverages of various heights in the to-be-recognized pattern can also be recognized, and the accuracy of image recognition is further improved.

[0083] The operation flow is combined into the specific embodiments below to illustrate the beverage cabinet image recognition process provided by the embodiments of the present application.

[0084] In an embodiment of the present disclosure, a camera is arranged below and below the beverage cabinet respectively, an angle sensor is arranged on the door of the beverage cabinet, and the beverage cabinet can also communicate with a cloud server.

[0085] Figure 3 is a flowchart of a process of acquiring a beverage cabinet image provided by an embodiment of the present disclosure. In combination with Figure 3 , the process of acquiring a beverage cabinet image includes:

[0086] Step 301: Determine whether the current angle collected by the angle sensor is greater than the set angle? If yes, execute step 302, otherwise, return to step 301.

[0087] Step 302: Start the camera to acquire two beverage images of the beverage cabinet.

[0088] Step 303: Send the two acquired beverage images to the cloud server.

[0089] In this way, the cloud server can acquire two beverage images of the beverage cabinet. Moreover, the ResNet deep learning algorithm model has been configured and saved in the cloud server.

[0090] Figure 4 is a flowchart of a process of a beverage cabinet image recognition method provided by an embodiment of the present disclosure. In combination with Figure 4 , the process of the beverage cabinet image recognition includes:

[0091] Step 401: Receive two beverage images sent by the beverage cabinet.

[0092] Step 402: image preprocessing including image distortion correction is performed on each beverage image to obtain a corresponding preprocessed image.

[0093] Step 403: based on the faster Rcnn target detection algorithm, each preprocessed image is detected to obtain beverage cabinet shelf position information and beverage size information of each layer in each preprocessed image.

[0094] Step 404: according to the beverage cabinet shelf position information and the beverage size information of each layer, each beverage image is spliced to obtain a to-be-recognized pattern.

[0095] Step 405: input the to-be-recognized pattern into the saved ResNet deep learning algorithm model to recognize the beverage information in the to-be-recognized pattern.

[0096] As can be seen, in the embodiment, when the door of the beverage cabinet is opened, two beverage images can be obtained by the camera and sent to the cloud server. Thus, the cloud server can perform image detection on each beverage image of the beverage cabinet and perform splicing processing to obtain a to-be-recognized pattern, and then recognize the beverage information of the beverage cabinet based on the deep learning algorithm. In this way, the automatic acquisition of the beverage information in the beverage cabinet is realized, and the intelligence of the beverage cabinet is improved. Moreover, the cloud server can reduce the memory occupation of the beverage cabinet by obtaining images from the beverage cabinet and performing corresponding processing and recognition, improve the running speed of the beverage cabinet, and the server can uniformly manage the beverage information of each beverage cabinet, further improving the dataization and intelligence of the beverage cabinet. Of course, the beverage cabinet shelf position information and the beverage size information of each layer detected by image detection are used for image splicing, which is not affected by factors such as light, shooting position, consistency of collection equipment, and image distortion, so that the image splicing of the beverage cabinet is good in integrity, and the accuracy of beverage detection and recognition is improved.

[0097] According to the above process for beverage cabinet image recognition, a device for beverage cabinet image recognition can be constructed.

[0098] Figure 5 is a structural schematic diagram of a device for beverage cabinet image recognition provided by the embodiment of the disclosure. As shown in Figure 5 the device for beverage cabinet image recognition includes an acquisition module 510, a detection module 520, a splicing module 530, and a recognition module 540.

[0099] The acquisition module 510 is configured to acquire at least two beverage images of a beverage cabinet.

[0100] The detection module 520 is configured to perform image detection on each beverage image to detect beverage cabinet shelf position information and beverage size information of each layer in each beverage image.

[0101] The splicing module 530 is configured to splice each beverage image according to the beverage cabinet layer position information and the size information of each layer of beverage to obtain a to-be-recognized pattern.

[0102] The recognition module 540 is configured to recognize beverage information in the to-be-recognized pattern based on a deep learning algorithm.

[0103] In some embodiments, the detection module 520 includes a preprocessing unit and a detection unit.

[0104] The preprocessing unit is configured to perform image preprocessing including image distortion correction on each beverage image to obtain a corresponding preprocessed image.

[0105] The detection unit is configured to perform detection on each preprocessed image based on a deep learning target detection algorithm to obtain beverage cabinet layer position information and size information of each layer of beverage in each preprocessed image.

[0106] In some embodiments, the preprocessing unit is specifically configured to determine a distortion coordinate (x'', y'') of a set image pixel in a current beverage image; determine a distortion correction coordinate (x', y') corresponding to the set image pixel and the distortion coordinate in the current beverage image through formula (1); and generate a corresponding current preprocessed image according to each distortion correction coordinate (x', y').

[0107]

[0108] wherein r 2 = x' + y', is a radial distortion coefficient, 2p2x'y' + p1(r 2 + 2y' 2 ) is a tangential distortion coefficient; k1-k6 are radial distortion parameters, p1, p2 are tangential distortion parameters, and s1, s2 are thin prism distortion parameters of the image acquisition device.

[0109] In some embodiments, the detection unit is specifically configured to input the normalized current preprocessed image into a VGG16 convolutional neural network for processing to obtain a current feature image; obtain a detection frame of the current feature image according to a Faster RCNN algorithm, and obtain beverage cabinet layer position information and size information of each layer of beverage in the current preprocessed image according to the detection frame.

[0110] In some embodiments, the splicing module 530 is specifically configured to determine a first height value HH between the current lowest shelf and the lower edge of the current beverage image according to the beverage cabinet shelf position information in the current beverage image; determine a second height h of the lowest layer of beverage according to the size information of each layer of beverage in the current beverage image; in the case that the first height value HH is less than the preset beverage height H, or the second height h is less than the preset beverage height H, determine the lower edge of the current beverage image as the current lowest shelf, and perform the cutting processing.

[0111] In some embodiments, the splicing module 530 is further configured to determine first beverage information of the beverage height less than the preset beverage height H in each beverage image according to the size information of each layer of beverage; and splice and sort each beverage image according to the first beverage information.

[0112] In some embodiments, the device further comprises:

[0113] The configuration module is configured to obtain sample pictures of at least two size types of the same beverage, wherein the height of the beverage in one size type is less than the preset beverage height H; and train the sample pictures based on the ResNet model to obtain a ResNet deep learning algorithm model.

[0114] It can be seen that, in the embodiment, the beverage cabinet image recognition device can perform image detection on each beverage image of the beverage cabinet obtained, and perform splicing processing to obtain a to-be-identified image, and then recognize the beverage information of the beverage cabinet based on the deep learning algorithm. In this way, the automatic acquisition of the beverage information in the beverage cabinet is realized, and the intelligence of the beverage cabinet is improved. Moreover, the image splicing is performed based on the beverage cabinet shelf position information and the size information of each layer of beverage detected by image detection, which is not affected by factors such as light, shooting position, consistency of collection equipment, and image distortion, so that the image splicing completeness of the beverage cabinet is good, and the accuracy of beverage detection and recognition is improved.

[0115] The embodiment of the present disclosure provides a device for beverage cabinet image recognition, which has the structure as shown in Figure 6 The device comprises:

[0116] The processor 100 and the memory 101 can also include a communication interface 102 and a bus 103. The processor 100, the communication interface 102, and the memory 101 can communicate with each other through the bus 103. The communication interface 102 can be used for information transmission. The processor 100 can call the logical instructions in the memory 101 to execute the method for beverage cabinet image recognition of the above-mentioned embodiments.

[0117] In addition, the logic instructions in the memory 101 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0118] The memory 101 as a computer readable storage medium can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiments of the present disclosure. The processor 100 executes the program instructions / modules stored in the memory 101, thereby performing functional applications and data processing, that is, implementing the method for beverage cabinet image recognition in the method embodiments described above.

[0119] The memory 101 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 101 can include a high-speed random access memory, and can also include a non-volatile memory.

[0120] The embodiments of the present disclosure provide a device comprising the above-described beverage cabinet image recognition apparatus.

[0121] The embodiments of the present disclosure provide a computer readable storage medium storing computer executable instructions, which are configured to execute the above-described beverage cabinet image recognition method.

[0122] The embodiments of the present disclosure provide a computer program product, which includes a computer program stored on a computer readable storage medium, and the computer program includes program instructions, which, when executed by a computer, cause the computer to execute the above-described beverage cabinet image recognition method.

[0123] The above-described computer readable storage medium can be a transitory computer readable storage medium or a non-transitory computer readable storage medium.

[0124] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, etc. various media that can store program codes, or a transitory storage medium.

[0125] The above description and drawings are illustrative of embodiments of the present disclosure and are not intended to be limiting. Other embodiments can include structural, logical, electrical, process, and other changes. Embodiments are illustrative of the many possible variations that are readily undertaken. Individual components and functions are optional unless explicitly required, and the order of operations can be varied. Portions and features of some embodiments can be included in, or substituted for, those of other embodiments. The scope of the present disclosure encompasses the entire scope of the following claims, and all available equivalents of the claims. When used in this application, the terms "first," "second," and the like, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without changing the meaning of the description, so long as all occurrences of the "first element" are renamed consistently and all occurrences of the "second element" are renamed consistently. The first element and the second element are both elements, but they are not necessarily the same element. Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. As used in the description of the embodiments and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. In addition, the term "comprises / comprising" and / or "comprises / comprising" when used in this application is taken to mean, for either the singular or plural forms, the stated features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Without more limitations, an element preceded by "comprises a" does not, without more limitations, preclude the existence of additional identical elements in the process, method, article, or apparatus including the recited element. In this document, each embodiment is highlighted by the differences from other embodiments. Identical or similar parts between embodiments can be mutually referred to. For the method, product, etc. disclosed by the embodiments, if it corresponds to the method part disclosed by the embodiments, the relevant part can be referred to the description of the method part.

[0126] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present disclosure. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0127] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units can only be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms. The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to implement the embodiments. In addition, each functional unit in the embodiments of the present disclosure can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit.

[0128] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

Claims

1. A method for beverage cabinet image recognition, characterized in that, The method comprises the following steps: acquiring at least two beverage images of a beverage cabinet; performing image detection on each of the beverage images to detect beverage cabinet shelf position information and size information of each layer of beverage in each of the beverage images; performing splicing processing on each of the beverage images according to the beverage cabinet shelf position information and the size information of each layer of beverage to obtain a to-be-recognized pattern; recognizing beverage information in the to-be-recognized pattern based on a deep learning algorithm; wherein the splicing processing on each of the beverage images comprises: determining a first height value HH between a current lowest shelf and a lower edge of a current beverage image according to beverage cabinet shelf position information in the current beverage image; determining a second height h of the lowest layer of beverage according to the size information of each layer of beverage in the current beverage image; in a case where the first height value HH is less than a preset beverage height H or the second height h is less than the preset beverage height H, determining the lower edge of the current beverage image as the current lowest shelf and performing cutting processing.

2. The method of claim 1, wherein, the image detection on each of the beverage images comprises: performing image preprocessing including image distortion correction on each of the beverage images to obtain a corresponding preprocessed image; detecting each of the preprocessed images based on a deep learning target detection algorithm to obtain beverage cabinet shelf position information and size information of each layer of beverage in each of the preprocessed images.

3. The method of claim 2, wherein, the image preprocessing including image distortion correction on each of the beverage images comprises: determining distortion coordinates of the set image pixels in the current beverage image , ) By formula (1), the distortion correction coordinates corresponding to the distortion coordinates of the set image pixels in the current beverage image are determined ); According to each of the aforementioned distortion correction coordinates ( ), generate the corresponding current preprocessed image; （1） wherein , is the radial distortion coefficient, k1~k6 are radial distortion parameters, p1, p2 are tangential distortion parameters, and s1, s2 are thin prism distortion parameters of an image acquisition device.

4. The method of claim 2, wherein, the detection of each of the preprocessed images based on the target detection algorithm comprises: inputting the current preprocessed image after normalization into a VGG16 convolutional neural network for processing to obtain a current feature image; obtaining a detection frame of the current feature image according to a Faster RCNN algorithm, and obtaining beverage cabinet shelf position information and size information of each layer of beverage in the current preprocessed image according to the detection frame.

5. The method of claim 1, wherein, the splicing processing on each of the beverage images according to the beverage cabinet shelf position information and the size information of each layer of beverage comprises: determining first beverage information with a beverage height less than a preset beverage height H in each of the beverage images according to the size information of each layer of beverage; performing splicing and sorting on each of the beverage images according to the first beverage information.

6. The method according to any one of claims 1 to 5, characterized in that, before the recognition of beverage information in the to-be-recognized pattern based on the deep learning algorithm, the method further comprises: acquiring sample pictures of at least two size types of the same beverage, wherein the height of the beverage in one size type is less than a preset beverage height H; training the sample pictures based on a ResNet model to obtain a ResNet deep learning algorithm model.

7. An apparatus for beverage cabinet image recognition, characterized by, The method comprises the following steps: an acquiring module configured to acquire at least two beverage images of a beverage cabinet; a detecting module configured to perform image detection on each of the beverage images to detect beverage cabinet shelf position information and size information of each layer of beverage in each of the beverage images; The splicing module is configured to splice each of the beverage images according to the beverage cabinet layer position information and the size information of each layer of beverage to obtain a to-be-recognized pattern; The recognition module is configured to recognize beverage information in the to-be-recognized pattern based on a deep learning algorithm. The splicing module is specifically configured to determine a first height value HH between a current lowest layer and a lower edge of a current beverage image according to beverage cabinet layer position information in the current beverage image, determine a second height h of the lowest layer of beverage according to the size information of each layer of beverage in the current beverage image, and determine the lower edge of the current beverage image as the current lowest layer and perform cutting processing in a case where the first height value HH is less than a preset beverage height H or the second height h is less than the preset beverage height H.

8. An apparatus for beverage cabinet image recognition, the apparatus comprising a processor and a memory having stored therein program instructions, the apparatus being characterized by: The processor is configured to execute the program instructions to perform the method for beverage cabinet image recognition according to any one of claims 1 to 6.

9. An apparatus, comprising: The device for beverage cabinet image recognition according to claim 7 or 8 is included.

Images