Order generation method, device and intelligent vending machine based on multi-level commodity detection

Abstract

The present invention belongs to the field of image processing technology, and solves the technical problem of low detection accuracy of existing intelligent vending machines, which leads to abnormal orders. It provides an order generation method, device and intelligent vending machine based on multi-level commodity detection, which obtains the target video of users shopping through the intelligent vending machine and the weight change information of the shelves in the commodity area, and divides each frame image of the target video into the first frame image and the corresponding non-first frame image according to the weight change information; then adds a basic confidence to each target in each frame image, and uses the target detection model to detect each frame image of the target video, and obtains the order information corresponding to each target commodity according to the detection result of each target combined with the corresponding basic confidence; uses the imaging distance difference between the first frame image and the non-first frame image to set the corresponding basic confidence, which can improve the target detection result by using images with close distance and many features, thereby improving the detection accuracy.

Description

[0001] This application is a divisional application of the invention patent application filed on November 9, 2021, with the invention name "Intelligent order generation method based on video segmentation processing based on weight changes in commodity areas" and application number 202111318652.6. Technical Field

[0002] The present invention relates to the field of image analysis technology, and in particular to an order generation method and device based on multi-level commodity detection and an intelligent vending machine. Background Art

[0003] With the continuous development of artificial intelligence technology, the sales methods of the retail industry have also undergone tremendous changes. Smart vending machines have been widely used in various places in the city, including stations, shopping malls, tourist attractions or department stores. Various types of smart vending machines can be found. Smart vending machines do not require special supervision, users can automatically place orders and check out, which greatly meets the shopping needs of users in special scenarios.

[0004] However, existing smart vending machines include fully-opening smart vending machines. When the door of the fully-opening smart vending machine is open, users can perform multiple pick-up and put-down operations in one shopping trip, and can select multiple items at a time and then settle the bill in a unified manner. This smart vending machine greatly facilitates the shopping needs of users. However, because this type of smart vending machine mainly relies on shopping videos to settle product orders, when users pick up and put down products, some features of the products will be blocked, which easily leads to false detection and abnormal order generation. In order to eliminate the problem of false detection of products due to occlusion, shopping videos are usually collected from multiple angles, and then each frame image of each video is detected, and the final product order is determined by comparing the detection results of each video. Because multiple videos need to be detected, the processor needs to have multi-threaded data processing capabilities, which requires greater computing power and cost, or single-threaded processing requires queuing and waiting, and the processing efficiency is low, which affects the user experience. Summary of the Invention

[0005] In view of this, the embodiments of the present invention provide an order generation method, device and intelligent vending machine based on multi-level commodity detection, which are used to solve the technical problem of low detection accuracy of existing intelligent vending machines, resulting in abnormal orders.

[0006] The technical solution adopted in the present invention is:

[0007] The present invention provides an order generation method based on multi-level commodity detection, the method comprising:

[0008] Obtain the target video and weight change information of the product area where the user shops through the smart vending machine;

[0009] According to the weight change information, each frame image of the target video is divided into a plurality of first frame images and a plurality of non-first frame images other than the first frame image according to the acquisition time;

[0010] Adding a basic confidence level to each target in each frame of the target video according to the first frame image and the non-first frame images;

[0011] The order information corresponding to each target product is obtained according to the basic confidence of each product and the detection results of each frame image of the target video input into the target detection network.

[0012] Preferably, dividing each frame image of the target video into a plurality of first frame images and a plurality of non-first frame images other than the first frame image according to the acquisition time according to the weight change information comprises:

[0013] Segmenting the target video according to each time information of the weight change information to obtain a plurality of target sub-videos;

[0014] Each frame image of each target sub-video is divided into the first frame image and the non-first frame image according to the acquisition time.

[0015] Preferably, the target video is segmented according to each time information of the weight change information to obtain a plurality of target sub-videos, including:

[0016] Segmenting the target video according to each time information of the weight change information to obtain a plurality of first videos;

[0017] According to the increase and decrease information of the weight change information, each of the first videos is divided into the target sub-videos corresponding to the product being put on the shelf and the product being taken off the shelf.

[0018] Preferably, the adding a basic confidence to each target in each frame image of the target video according to the first frame image and the non-first frame images includes:

[0019] Determining, based on the image information of the first frame image, the positioning information of the commodity area to which each commodity in the first frame image belongs;

[0020] According to the positioning information, a basic confidence level is added to each target in each frame image of the target video.

[0021] Preferably, obtaining the target video and weight change information of the commodity area purchased by the user through the smart vending machine includes:

[0022] Obtain a status video of the smart vending machine, dividing the product placement area of the smart vending machine into multiple virtual product areas along the shelf arrangement direction of the smart vending machine, and the current status of the smart vending machine;

[0023] Analyze each frame of the status video to obtain status information of the smart vending machine cabinet door, wherein the status information includes an open state and a closed state;

[0024] Control each camera to collect video data of the corresponding commodity area according to the on state, and control each camera to stop collecting video data of the corresponding commodity area according to the off state, to obtain each basic video;

[0025] Physically splicing each frame image of each basic video according to the corresponding frame image of the acquisition time sequence to obtain the target video;

[0026] The physical stitching means that the stitched image is the sum of the sizes of all images involved in the stitching.

[0027] Preferably, the step of obtaining order information corresponding to each target product by inputting the basic confidence of each product into a target detection network in combination with the detection results of each frame image of the target video includes:

[0028] Using the target detection network to perform target detection on each frame of the target video to obtain basic product information of each product;

[0029] According to the confidence of the basic product information of each product, combined with the basic confidence of each product, the product information including the target confidence of each product is obtained;

[0030] Deduplication of the products according to the product location information of each product information to obtain target products;

[0031] According to the product information of each target product, order information corresponding to each target product is output.

[0032] Preferably, deduplicating each of the products according to the product location information of each of the product information to obtain each target product includes:

[0033] Partitioning each frame image of the target video into image regions of each viewing angle to obtain image sub-regions corresponding to the image of each viewing angle;

[0034] Obtaining, based on the image sub-regions and the imaging size information corresponding to the respective products, a relative relationship between the imaging size information of the respective products belonging to the same image sub-region at different viewing angles;

[0035] The products are deduplicated according to the relative relationship of the imaging size information of the products to obtain the target products.

[0036] The present invention also provides an order generation device based on multi-level commodity detection, the device comprising:

[0037] Video acquisition module: used to obtain target videos of users shopping at smart vending machines and weight change information of product areas;

[0038] A video segmentation module is configured to divide each frame image of the target video into a plurality of first frame images and a plurality of non-first frame images other than the first frame image according to the acquisition time based on the weight change information;

[0039] A data processing module is configured to add a basic confidence level to each target in each frame of the target video according to the first frame image and the non-first frame images;

[0040] Order generation module: used to obtain order information corresponding to each target product based on the basic confidence of each product and the detection results of each frame image of the target video input into the target detection network.

[0041] The present invention also provides an intelligent vending machine, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, wherein when the computer program instructions are executed by the processor, any of the above-mentioned methods is implemented.

[0042] The present invention also provides a medium having computer program instructions stored thereon, which implement any of the above methods when the computer program instructions are executed by a processor.

[0043] In summary, the beneficial effects of the present invention are as follows:

[0044] The present invention provides an order generation method, device and intelligent vending machine based on multi-level commodity detection, which obtains the target video of the user shopping through the intelligent vending machine and the weight change information of the shelves in the commodity area, and divides each frame image of the target video into the first frame image and the corresponding non-first frame image according to the weight change information; then adds a basic confidence to each target in each frame image, and uses the target detection model to detect each frame image of the target video, and obtains the order information corresponding to each target commodity according to the detection result of each target combined with the corresponding basic confidence; uses the imaging distance difference between the first frame image and the non-first frame image to set the corresponding basic confidence, which can improve the target detection result by using images with close distance and many features, thereby improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work, and these are all within the scope of protection of the present invention.

[0046] Figure 1 This is a flow chart of the method for intelligently generating orders for products through multi-perspective identification in Example 1;

[0047] Figure 2 This is a schematic diagram of the structure of the smart vending machine with multiple cameras of different viewing angles in Example 1;

[0048] Figure 3 This is a schematic diagram of the process of obtaining a physically spliced target video in Example 1;

[0049] Figure 4 This is a schematic diagram of the process of obtaining product information in Example 1;

[0050] Figure 5 This is a schematic diagram of the process of removing duplicate products in Example 1;

[0051] Figure 6 This is a flow chart of the intelligent order generation method for processing video segments based on weight changes in commodity areas in Example 2;

[0052] Figure 7 Schematic diagram of the process of stitching target video with base video in Example 2;

[0053] Figure 8 Schematic diagram of the process of obtaining a target sub-video in Example 2;

[0054] Figure 9 This is a schematic diagram of the process of generating order information in Example 2;

[0055] Figure 10 This is a flow chart of the device for intelligently generating orders for products with multi-view recognition in Example 3;

[0056] Figure 11 This is a flow chart of an intelligent order generation device for processing video segments based on weight changes in commodity areas in Example 4;

[0057] Figure 12 Schematic diagram of the structure of the automatic settlement system including the intelligent vending machine in Example 5;

[0058] Figure 13 This is a schematic diagram of the structure of the intelligent vending machine in Example 6;

[0059] Figures 1 to 13 Reference numerals:

[0060] 1. Cabinet body; 11. Shelves; 12. Merchandise area; 2. Cabinet doors; 3. Camera. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further limitations, elements defined by the phrase "comprising..." do not preclude the presence of other identical elements in the process, method, article, or apparatus comprising the elements. The various features of the present invention and its embodiments may be combined with each other if there is no conflict, and all are within the scope of protection of the present invention.

[0062] Example 1

[0063] Existing fully-opening smart vending machines have the advantages of being convenient for users to purchase multiple items at one time and change items multiple times during one shopping trip. At the same time, they can quickly generate orders when users complete a complex shopping process and quickly settle accounts through autonomous settlement. Compared with the existing methods that can only purchase one item by scanning a code once and cannot be re-selected after purchase, fully-opening smart vending machines have the advantages of simple operation and greater user autonomy in shopping. However, because fully-opening smart vending machines allow users to purchase multiple items in one shopping trip and can put and take items off the shelves multiple times, the occlusion problem will cause a large number of items to be detected differently before and after the same item is put on or taken off the shelves due to different occlusion positions, resulting in abnormal orders that affect user experience and merchant reputation.

[0064] The present invention is based on a feasibility study of obtaining shopping videos of users shopping from smart vending machines from multiple angles. By setting cameras in the product area of the smart vending machine to monitor the product area from different directions in real time, combining the shopping videos shot by multiple cameras, and then obtaining the user's shopping order information through image splicing, comparative analysis, etc., and then automatically settling the bill through the server, the user's shopping experience is improved, while reducing the manual settlement process.

[0065] For details, see Figure 2 , Figure 2 The diagram is a structural diagram of a fully-open smart vending machine, which includes a cabinet body 1 and a cabinet door 2. The cabinet body 1 and the cabinet door 2 are rotatably connected, and when the cabinet door 2 is in a closed state relative to the cabinet body 1, the cabinet door 2 covers all commodity areas of the cabinet body 1 where commodities are placed, i.e., commodities in the cabinet body cannot be taken out. When the cabinet door 2 is opened, all commodities in the cabinet body 1 are displayed in front of the user. The user can select any commodity in a shopping mall, and can also select multiple commodities. The user can take out the selected commodity or put back the commodity that needs to be put back after selection. A shelf 11 is provided in the cabinet body 1, and the shelf 11 can be a shelf that divides the cabinet body 1 into multiple commodity areas 12. A camera is provided in each commodity area inside the cabinet body 1, so that a shopping video of a user shopping from the smart vending machine can be obtained from multiple angles, thereby avoiding the problem that the shopping video collected from a single angle is not reliable due to occlusion. Figure 2 The smart vending machine shown is equipped with multiple cameras on the left and right internal walls of the vending machine so that shopping videos can be collected from the same product area in relative viewing directions, thereby improving the reliability of the video data.

[0066] See Figure 1 , Figure 1 A flowchart of a method for intelligently generating orders for products through multi-perspective product recognition is provided. The method includes:

[0067] S10: Obtain a target video of the product area, wherein the target video is composed of a video stream obtained by physically splicing frames corresponding to several basic videos, each of which is composed of images of the same event occurring in the product area captured from a different perspective, and each basic video captures the same event from a different perspective;

[0068] Specifically, cameras are installed at different positions of the smart vending machine to obtain video data of the product area of the smart vending machine from different perspectives. When the user starts shopping at the smart vending machine, each camera obtains basic video of the user taking or putting back the product from different angles, and the frames of the basic video obtained by different cameras are physically spliced according to the acquisition sequence to obtain the target video composed of spliced images; wherein the same event is the entire shopping process of the user.

[0069] It should be noted that physical stitching is to stitch two images into one image, and the stitched image is the sum of the sizes of the images involved in the stitching; at the same time, the physical stitching of each frame image of different videos is: the first frame image of the first video, the first frame image of the second video... the first frame image of the Nth video are stitched together, the second frame image of the first video, the second frame image of the second video... the second frame image of the Nth video are stitched together, and so on, the nth frame image of the first video, the nth frame image of the second video... the nth frame image of the Nth video are stitched together to obtain the target video.

[0070] In one embodiment, see Figure 3 , the S10 includes:

[0071] S101: Dividing the merchandise area of the smart vending machine into multiple virtual merchandise areas along the arrangement direction of the shelves of the smart vending machine;

[0072] Specifically, if a smart vending machine has multiple layers of shelves, the area where the merchandise is placed is divided into multiple product areas. Each product area includes at least one layer of shelves. The camera's field of view is set along the arrangement of the shelves. For example, if the shelves of the smart vending machine include multiple layers from top to bottom, the cameras are respectively installed on the left and right side walls of the smart vending machine, with each camera's field of view from top left to bottom right, top right to bottom left, or top to bottom. Cameras installed on different sides of the same product area are installed at the same height.

[0073] In one embodiment, the S101 includes:

[0074] S1011: Dividing the merchandise area of the smart vending machine into an upper merchandise area and a lower merchandise area along a camera's viewing angle from top to bottom;

[0075] S1012: A camera is provided on the left and right sides of the upper commodity area and on the left and right sides of the lower commodity area respectively;

[0076] The viewing angle of the camera on the left is from the upper left corner to the lower right corner, and the viewing angle of the camera on the right is from the upper right corner to the lower left corner.

[0077] Specifically, in a preferred embodiment, the shelves of the smart vending machine are divided into an upper merchandise area and a lower merchandise area, and a camera is set on the left and right side walls of the upper and lower merchandise areas. The left and right cameras in the upper merchandise area can collect video data of the entire area, and the left and right cameras in the lower merchandise area can only collect video data within the range of the lower merchandise area.

[0078] It should be noted that dividing the shelf into two upper and lower merchandise areas, and setting up a pair of cameras in each merchandise area, can ensure that shopping video data is obtained from four directions of up, down, left and right during a shopping event, thereby improving data reliability. This method can save costs and control the size of each frame image of the target video, reducing the amount of data processing.

[0079] S102: Obtain basic videos within the viewing angle range captured by cameras disposed opposite to each other in each commodity area. Specifically, each camera obtains a video stream of the corresponding area to obtain a video of the user taking or putting the commodity back.

[0080] S103: physically splicing the frame images of the basic videos one by one according to the corresponding frame images in the acquisition time sequence to obtain the target video;

[0081] The physical stitching means that the stitched image is the sum of the sizes of all images involved in the stitching.

[0082] Specifically, the frames of the different basic videos are physically spliced together. For the specific splicing method, refer to the above method to obtain the final target video.

[0083] In one embodiment, the S10 includes:

[0084] S105: Obtaining the frame rate and number of cameras used to collect video data;

[0085] S106: Determine, based on the frame rate and the number of cameras, the interval at which each camera starts collecting video data of the corresponding product area;

[0086] S107: Controlling each camera to acquire corresponding basic videos according to each interval time;

[0087] Specifically, the frame rates of the cameras used to collect video data are the same, such as 20 frames per second; the time when each camera starts collecting video data is determined according to the number and frame rate of the cameras, and there is a time interval between the start times of collecting video data between each camera or each group of cameras, wherein the preferred interval time is an integer multiple of the time difference corresponding to two adjacent frames of images, such as: including 4 cameras, the start time of collection of each camera is separated by 1 / 4 of the time corresponding to the frame rate, or the 4 cameras are divided into two groups, and the interval time when each group of cameras starts collecting video data is 1 / 2 of the time corresponding to the frame rate; thereby indirectly increasing the image frame rate, ensuring that image information of the product area at more moments is collected, so as to improve the accuracy of detection.

[0088] S108: Physically splicing the frame images of each of the basic videos corresponding to the acquisition time sequence one by one to obtain the target video.

[0089] S11: Input each frame image of the target video into the target detection network to perform target detection and obtain product information of each product;

[0090] Specifically, the spliced image frames are fed into a target detection network for target detection to obtain product information of each detected target, where the product information includes at least one of the following: product location information in the image, area information of the detection frame, product category information, and confidence level.

[0091] In one embodiment, see Figure 4 , the S11 includes:

[0092] S111: dividing each frame image of the target video into a first frame image and non-first frame images other than the first frame image according to the acquisition time;

[0093] In one embodiment, the S111 includes:

[0094] S1111: Detect the target quantity of each frame image of the target video to determine each frame image corresponding to a product change;

[0095] S1112: Divide the target video into multiple target sub-videos according to the image frames in which the product changes;

[0096] S1113: Divide each frame image of each target sub-video into the first frame image and the non-first frame image according to the acquisition time.

[0097] S112: Determine, based on the image information of the first frame image, the location information of the commodity area to which each commodity in the first frame image belongs;

[0098] Specifically, each frame image of the target video is divided into a first frame image and non-first frame images other than the first frame image, wherein the first frame image includes but is not limited to the first frame image of the complete target video, and can also be the target video divided into multiple video segments, and the first frame image is the first frame image of each video segment; the first frame image is subjected to preliminary target detection to determine which commodity area the target comes from, and the video data captured by the camera in the commodity area where the commodity comes from is recorded as the positioning video of the commodity.

[0099] S113: adding a basic confidence level to each target in each frame of the target video according to the positioning information;

[0100] Specifically, after determining the positioning information of the product, a basic confidence level is added to each target in each frame image of the positioning video corresponding to the positioning information. For example, the smart vending machine is divided into multiple product areas, and each product area is equipped with a corresponding camera. It should be noted that when taking or putting products, not only the camera that takes out the corresponding product area or puts it into the corresponding product area will collect the corresponding video data, but the cameras in other product areas may also collect the corresponding video data. For example, the product area of the smart vending machine is divided into the upper product area and the lower product area. When taking products from the lower product area, the products are both in the camera position of the upper product area. The video also exists in the video of the camera in the lower commodity area. Because the product belongs to the lower commodity area, the image size of the product in the video data of the camera in the lower commodity area is larger, which is conducive to improving the detection accuracy. Therefore, a basic confidence can be added to the target in each frame image of the video taken by the camera in the lower commodity area, recorded as the first basic confidence, and a basic confidence can be added to the target in each frame image of the video corresponding to the camera in the upper commodity area, recorded as the second basic confidence, where the second basic confidence is less than the first basic confidence; similarly, when the product belongs to the upper commodity area, the second basic confidence is greater than the first basic confidence.

[0101] In a preferred embodiment, the S113 includes:

[0102] S1131: Obtaining a boundary line of the product movement corresponding to the product confidence enhancement;

[0103] S1132: Segment the target video according to different state areas of the commodity located on the boundary line in adjacent image frames to obtain a first video segment with enhanced confidence and a second video segment with normal confidence;

[0104] S1133: In combination with the positioning video, add a basic confidence level to each target in each frame image of the first video segment corresponding to the product area to which the target product belongs.

[0105] Specifically, after the product is taken out of the product area, as the distance from the camera increases, the imaging size of the product in the image decreases, so the detection accuracy will decrease. Therefore, each frame image close to the camera area is used as the key detection object. Therefore, the target video is divided into a first video segment and a second video segment using a boundary line, and the basic confidence of each target in the image area of the positioning video in each frame image of the first video segment is increased, or, the first basic confidence of each target in the image area of the positioning video in each frame image of the first video segment is increased, and the second basic confidence of each target in the image area of the positioning video in each frame image of the first video segment that does not belong to the image area of the positioning video is increased, wherein the first basic confidence is greater than the second basic confidence.

[0106] S114: Performing target detection on each frame of the target video using the target detection network to obtain basic product information of each product;

[0107] S115: The basic confidence of each product is superimposed on the confidence of the basic product information of each product to obtain the product information including the target confidence of each product.

[0108] Specifically, each frame image of the target video is sent to the target detection network for detection to obtain basic product information of each target in each frame image, wherein the basic product information includes at least one of the following: product category, confidence and product location information of the detection frame representing the detected product. The confidence of each target detected this time is recorded as the actual confidence, and then the actual confidence of each target belonging to the first video segment is added to the basic confidence to obtain the target confidence of each target in the first video segment; the actual confidence of each target in the second video segment is used as the final target confidence, thereby obtaining the product information of each target composed of the target confidence.

[0109] S12: Deduplication of the products according to the product location information of each product information to obtain target products;

[0110] Specifically, based on the position information of the detection frame used to represent the detected target in the product information, it is determined which of the videos shot by different cameras are the same product, thereby achieving product deduplication; specific methods for product deduplication include but are not limited to using a classifier to detect the same product, and using the prior relationship between the product imaging position and imaging size in the same frame image in different videos to distinguish the same product and achieve product deduplication.

[0111] In one embodiment, see Figure 5 , the S12 includes:

[0112] S121: Acquire multiple positive samples and multiple negative samples, wherein each positive sample is an object belonging to the same product and appearing at a different position in the image after stitching images from different perspectives, and each negative sample is an object belonging to different products and appearing at a different position in the image after stitching images from different perspectives;

[0113] Specifically, each camera on the smart vending machine is controlled to obtain training videos of multiple shopping training times, and each target in each frame image of the training video is manually labeled. The targets with different position information belonging to the same product in the image area captured by different cameras in each frame image are recorded as positive samples, and the targets with other position information are recorded as negative samples. That is, two products are taken at a time, recorded as product A and product B. Assuming that there are four cameras with different perspectives, and in any frame all four cameras capture the image of product A and product B, then the corresponding target frame image will have 4 positive samples consisting of product A and 4 positive samples consisting of product B, as well as a negative sample between product A and product B.

[0114] S122: Inputting the samples including the positive samples and the negative samples into a support vector machine for training, thereby obtaining a product deduplication classifier capable of distinguishing whether products from different perspectives are the same product based on product location information;

[0115] Specifically, a manually labeled sample set is used for training to obtain a classifier that can distinguish whether products are the same product based on their location information. The classifier is then used to deduplicate each frame of the target video, so that the targets detected in each frame are independent products.

[0116] S123: Deduplication is performed using the product deduplication classifier according to the product location information of each product information to obtain each target product;

[0117] The samples are images obtained by physically stitching together frames of images captured by cameras on the smart vending machine.

[0118] In one embodiment, the S12 includes:

[0119] Step 1: partition each frame image of the target video into image regions of each viewing angle to obtain image sub-regions corresponding to the image of each viewing angle;

[0120] Specifically, each frame image of the target video is obtained by physically stitching together the image areas of each frame image captured by each camera. Therefore, each image area of each frame image of the target video is divided into multiple image sub-areas. For example, the target video is stitched together from the image frames of four videos. Therefore, each frame image of the target video includes four image areas, which are recorded as the upper left image area, the upper right image area, the lower left image area, and the lower right image area. Then, each image area is divided into multiple image sub-areas.

[0121] Step 2: Based on the image sub-regions and the imaging size information corresponding to the respective products, a relative relationship of the imaging size information of the respective products belonging to the same image sub-region at different viewing angles is obtained;

[0122] Specifically, the image sub-areas and imaging sizes corresponding to the imaging positions of the goods detected in images of different perspectives are compared to determine the relative relationship of the imaging sizes of targets belonging to images of different perspectives and the same image sub-area, such as: targets are detected in the lower right corners of the four image areas, wherein the image overlap of the targets in the upper left image area and the upper right image area is very high and greater than the overlap threshold, the target imaging range of the lower left image area belongs to the target imaging range of the upper left image area, and the target imaging range of the lower right image area belongs to the target imaging range of the upper right image area, it can be determined that the targets detected in the four image areas are the same product; if the image overlap of the targets in the upper left image area and the upper right image area is very low and less than the overlap threshold, the target imaging range of the lower left image area belongs to the target imaging range of the upper left image area, and the target imaging range of the lower right image area belongs to the target imaging range of the upper right image area, it can be determined that the targets detected in the upper left and lower left image areas are the same product, and the targets detected in the upper right and lower right image areas are the same product; including but not limited to the above situations, they are not listed here one by one.

[0123] Step 3: De-duplicate products based on the relative relationship of the imaging size information of each product to obtain the target products.

[0124] Specifically, after determining multiple detection targets corresponding to the same product in each frame image, duplicate removal is performed, and the target with high confidence can be selected and retained as the detection result of the same product.

[0125] S13: Outputting order information corresponding to each target product according to the product information of each target product;

[0126] The product information includes at least one of the following: product category, confidence level, and product location information of a detection frame representing the detected product.

[0127] In one embodiment, before S10, the method further includes:

[0128] S01: Obtain the video of the current state of the smart vending machine captured by the third camera in real time;

[0129] Specifically, the smart vending machine is also provided with a third camera, which is used to detect whether the smart vending machine is turned on or off. The third camera can be turned on in real time or after the user makes a shopping request.

[0130] S02: Analyze each frame of the video of the current state of the smart vending machine to determine whether the door of the smart vending machine is in an open or closed state;

[0131] S03: When it is detected that the door of the smart vending machine is in an open state, the first main camera, the first sub-camera, the second main camera and the second sub-camera for collecting video information corresponding to the product area are controlled to be turned on;

[0132] S04: When it is detected that the cabinet door of the smart vending machine is in a closed state, the first main camera, the first sub-camera, the second main camera and the second sub-camera for collecting video information corresponding to the commodity area are controlled to be turned off.

[0133] Specifically, when the user performs automatic shopping, each frame image of the vending machine status video is analyzed to determine the status of the vending machine door. When it is detected that the vending machine door is open, the first main camera, the first sub-camera, the second main camera and the second sub-camera are turned on to obtain video data of the commodity area to obtain each basic video; when it is detected that the vending machine door is closed, the first main camera, the first sub-camera, the second main camera and the second sub-camera are turned off.

[0134] The multi-perspective commodity recognition and intelligent order generation method of this embodiment is adopted. By obtaining videos of the commodity area from different perspectives and physically stitching them together, a target video is obtained. Then, target detection is performed on each frame image of the target video and the same commodity within the same frame image is deduplicated to obtain the target commodity for producing order information. By obtaining videos of shopping events from different perspectives, this method can prevent order anomalies caused by obstruction of commodities, thereby improving detection accuracy and user experience.

[0135] Example 2

[0136] In Example 1, video data of the product area of the smart vending machine is obtained from different perspectives, and then the frames of the different videos are spliced together to obtain the target video. The frames of the target video are analyzed to obtain product order information. However, during a shopping event, users may perform complex operations such as repeated selection or repeated switching, resulting in multiple pick-up and drop-down events, which often leads to false detection or mixed detection of products with high similarity, affecting detection accuracy. Therefore, Example 2 of the present invention further improves the method for intelligently generating orders for multi-perspective product recognition based on Example 1; please refer to Figure 6 , the method comprising:

[0137] S20: Obtaining a target video of the product area and weight change information of the product area;

[0138] Specifically, the target video of the product area is the image data collected by the camera of the smart vending machine during a shopping process. The target video can be the video data corresponding to the product area collected by a camera, or the video data collected by multiple cameras, or the video data collected by multiple cameras after being processed by splicing, fusion, etc.; the weight change information of the product area includes weight increase and decrease information and time information. Only using the weight increase and decrease information can quickly determine the action of putting the product on the shelf or taking the product off the shelf, thereby providing guidance for the neural network to perform target detection, and even combined with a small amount of image analysis to directly determine whether the user finally took the product. For example: if the weight decrease is detected at the first moment, it means that the user took the product. Through image analysis, it is found that the user only took one product. At the second moment, the weight increase is detected. If no other weight change information is detected during this period, it can be determined that the user put the product taken out for the first time directly back on the shelf, and then the video data from the first moment to the second moment can be deleted to reduce the subsequent data processing volume.

[0139] In one embodiment, see Figure 7 , the S20 includes:

[0140] S201: Acquire basic videos of the product area captured from different perspectives;

[0141] Specifically, cameras are set up at different positions of the smart vending machine to obtain basic videos of the product area of the smart vending machine from different perspectives, which can provide more reliable image data for target detection during shopping and improve the accuracy of target detection.

[0142] In one embodiment, the step S201 includes:

[0143] S2011: Dividing the product placement area of the smart vending machine into multiple virtual product areas along the shelf arrangement direction of the smart vending machine;

[0144] S2012: Obtain basic videos within the viewing angle range captured by cameras disposed opposite to each other in each commodity area.

[0145] Specifically, a smart vending machine has multiple layers of shelves. The area where the merchandise is placed is divided into multiple product areas, each of which includes at least one layer of shelves. The camera's field of view is set along the arrangement of the shelves. For example, if the shelves of the smart vending machine include multiple layers from top to bottom, the cameras are respectively set on the left and right side walls of the smart vending machine, with each camera's field of view running from top left to bottom right, or from top right to bottom left, or from top to bottom. Cameras located on different sides of the same product area are installed at the same height. When a user makes a purchase, each camera is turned on to capture video of the purchase, thereby generating basic videos of the purchase from different perspectives.

[0146] In one embodiment, the step S201 includes:

[0147] S2014: Obtaining the frame rate and number of cameras used to collect video data;

[0148] S2015: Determine, based on the frame rate and the number of cameras, the interval at which each camera starts collecting video data of the corresponding product area;

[0149] S2016: Controlling each of the cameras to obtain corresponding basic videos according to each of the interval times.

[0150] Specifically, the frame rate of each camera used to collect video data is the same, such as 20 frames per second; the time when each camera starts collecting video data is determined according to the number and frame rate of the cameras, and there is a time interval between the start time of collecting video data between each camera or each group of cameras, wherein the preferred interval time is an integer multiple of the time difference corresponding to two adjacent frames of images, such as: including 4 cameras, the start time of collecting video data of each camera is separated by 1 / 4 of the time corresponding to the frame rate, or the 4 cameras are divided into two groups, and the interval time for each group of cameras to start collecting video data is 1 / 2 of the time corresponding to the frame rate; thereby indirectly increasing the image frame rate, ensuring that image information of the product area at more moments is collected, so as to improve the accuracy of detection; after detecting the start of shopping, each camera obtains the video data of the corresponding area according to its own start time to obtain each basic video, wherein the signal for starting shopping is the user verifying through an identification code, such as: a QR code, a barcode, etc., or after the cabinet door of the smart vending machine is opened. The specific signal for starting shopping is not limited here.

[0151] S202: Physically splicing the frame images of the basic videos one by one according to the corresponding frame images in the acquisition time sequence to obtain the target video;

[0152] The physical stitching means that the stitched image is the sum of the sizes of all images involved in the stitching.

[0153] Specifically, when a user is shopping, each camera captures basic videos of the user taking or putting back the goods from different angles, and the frames of the basic videos captured by different cameras are physically spliced together to obtain a target video composed of the spliced images; wherein the same event is the entire shopping process of the user.

[0154] It should be noted that physical stitching is to stitch two images into one image, and the stitched image is the sum of the sizes of the images involved in the stitching; at the same time, the physical stitching of each frame image of different basic videos is: the first frame image of the first video, the first frame image of the second video... the first frame image of the Nth video are stitched together, the second frame image of the first video, the second frame image of the second video... the second frame image of the Nth video are stitched together, and so on, the nth frame image of the first video, the nth frame image of the second video... the nth frame image of the Nth video are stitched together to obtain the target video.

[0155] S21: Segmenting the target video according to the weight change information to obtain multiple target sub-videos;

[0156] Specifically, the weight change information of the commodity area is detected in real time. When a weight change is detected, it is considered that there is an action of picking up and putting in the commodity. The target video between this weight change and the previous weight change is used as a target sub-video, thereby dividing the target video into multiple target sub-videos; target detection can be performed on each target sub-video to obtain multiple commodity information corresponding to each target sub-video, thereby improving the accuracy of commodity orders.

[0157] In one embodiment, see Figure 8 , the S21 includes:

[0158] S211: Segment the target video into a plurality of first videos according to the time information of the weight change information;

[0159] S212: According to the increase and decrease information of the weight change information, each of the first videos is divided into the target sub-videos corresponding to the product being put on the shelf and the product being taken off the shelf.

[0160] Specifically, when a weight change is detected in the commodity area, the current image frame is determined based on the time information of the weight change to obtain a target sub-video, and then the target sub-video is determined to be a commodity placing video or a commodity taking video based on the increase or decrease information of the weight change information.

[0161] In one embodiment, the S212 includes:

[0162] S2121: Obtain the boundary line used to define whether a product is on the shelf or off the shelf;

[0163] S2122: Based on the different status areas of the commodities located on the boundary line in adjacent image frames and the corresponding increase and decrease information of the weight change information, each of the first videos is divided into the target sub-videos corresponding to commodity listing and commodity removal.

[0164] Specifically, a virtual boundary line is set in the door area of the smart vending machine. The boundary line is used to determine the taking or putting of goods in combination with weight change information. This is because when users select goods, they may take and put goods in the same area multiple times in a very short time. The picture at this time belongs to the interior of the smart vending machine, and these goods are severely obscured. If the video is segmented only by weight change, a large number of extremely short videos will be generated. Analyzing these extremely short videos separately not only increases the amount of calculation, but also has little meaning and may even increase the detection error rate. By dividing each video segment based on whether the goods leave or enter the boundary line, the accuracy of detection can be improved.

[0165] S22: Inputting each frame image of each target sub-video into an object detection network to perform object detection, and obtaining product information corresponding to each target sub-video;

[0166] S23: Output order information according to the product information corresponding to each target sub-video.

[0167] In one embodiment, see Figure 9 , the S23 includes:

[0168] S231: Deduplicating the same product in each frame image based on the product location information of the product information to obtain target products after deduplication;

[0169] For details, please refer to the deduplication method in the embodiment, which will not be described here in detail.

[0170] In one embodiment, the S231 includes:

[0171] S2311: Acquire multiple positive samples and multiple negative samples, wherein each positive sample is an object belonging to the same product and appearing at a different position in the image after stitching images from different perspectives, and each negative sample is an object belonging to a different product and appearing at a different position in the image after stitching images from different perspectives;

[0172] S2312: Inputting the samples including the positive samples and the negative samples into a support vector machine for training, thereby obtaining a product deduplication classifier capable of distinguishing whether products viewed from different perspectives are the same product based on product location information;

[0173] S2313: Deduplication is performed using the product deduplication classifier based on the product location information of each product information to obtain each target product;

[0174] The samples are images obtained by physically stitching together frames of images captured by cameras on the smart vending machine.

[0175] Specifically, for product deduplication, refer to the method in Example 1, which will not be described in detail here.

[0176] In one embodiment, the S231 includes:

[0177] Step 1: partition each frame image of the target video into image regions of each viewing angle to obtain image sub-regions corresponding to the image of each viewing angle;

[0178] Step 2: Based on the image sub-regions and the imaging size information corresponding to the respective products, a relative relationship of the imaging size information of the respective products belonging to the same image sub-region at different viewing angles is obtained;

[0179] Step 3: De-duplicate products based on the relative relationship of the imaging size information of each product to obtain the target products.

[0180] Specifically, after determining multiple detection targets corresponding to the same product in each frame image, duplicate removal is performed, and the target with high confidence can be selected and retained as the detection result of the same product.

[0181] Specifically, for product deduplication, refer to the method in Example 1, which will not be described in detail here.

[0182] S232: Outputting the order information corresponding to each target product according to the product information of each target product.

[0183] The intelligent order generation method based on video segmentation processing of weight changes in the commodity area of this embodiment is adopted. By real-time collection of weight change information of the commodity area during the user's shopping process, the target shopping video is segmented according to the weight change information to obtain multiple target sub-videos, and then each frame image of each target sub-video is used to perform target detection using a target detection network, and finally the commodity information purchased by the user is obtained, thereby generating shopping order information; this method divides the complete target video into multiple target sub-videos for target detection based on weight changes, which can avoid the mutual influence of taking out or putting in events, and improve detection accuracy and user experience.

[0184] Example 3

[0185] Example 3 of the present invention further provides a device for intelligently generating orders for goods from multiple perspectives based on the methods of Example 1 to Example 2. Figure 10 ,include:

[0186] Video capture module: used to obtain a target video of the product area, wherein the target video is composed of a video stream obtained by physically splicing the frames corresponding to several basic videos. Each basic video is composed of images of the same event occurring in the product area captured from a different perspective.

[0187] Target detection module: used to input each frame image of the target video into the target detection network for recognition, and obtain the product information of each product;

[0188] Target processing module: used for removing duplicates from each product according to the product location information of each product information to obtain target products;

[0189] An order generation module is configured to output order information corresponding to each target product according to the product information of each target product;

[0190] The product information includes at least one of the following: product category, confidence level corresponding to each product category, and product location information of a detection frame representing a detected product.

[0191] The order generation device based on multi-view image analysis of this embodiment obtains videos of the product area from different viewpoints and physically stitches them together to obtain a target video. Then, target detection is performed on each frame of the target video and the same product within the same frame is deduplicated to obtain the target product for producing order information. By obtaining videos of shopping events from different viewpoints, this method can prevent order anomalies caused by obstruction of products, thereby improving detection accuracy and user experience.

[0192] It should be noted that the device also includes the remaining technical solutions described in Examples 1 and 2, which will not be repeated here.

[0193] Example 4

[0194] Example 4 of the present invention further provides an intelligent order generation device based on the method of Example 1 to Example 2 for processing video segments based on the weight change of commodity areas, see Figure 11 ,include:

[0195] Video acquisition module: used to obtain the target video of the product area and the weight change information of the product area;

[0196] Video segmentation module: used for segmenting the target video according to the weight change information to obtain multiple target sub-videos;

[0197] Data processing module: used for inputting each frame image of each target sub-video into the target detection network to perform target detection, and obtain the commodity information corresponding to each target sub-video;

[0198] Order generation module: used to output order information based on the product information corresponding to each target sub-video.

[0199] The intelligent order generation device based on the weight change of the commodity area in this embodiment collects the weight change information of the commodity area in real time during the user's shopping process, segments the shopping target video according to the weight change information, obtains multiple target sub-videos, and then uses the target detection network to perform target detection on each frame image of each target sub-video, and finally obtains the product information purchased by the user, thereby generating shopping order information; this method divides the complete target video into multiple target sub-videos for target detection based on weight change, which can avoid the mutual influence of taking out or putting in events, and improve detection accuracy and user experience.

[0200] It should be noted that the device also includes the remaining technical solutions described in Example 4, which will not be repeated here.

[0201] Example 5

[0202] The present invention provides an automatic settlement system for intelligent vending machines. Figure 12 The automatic settlement system includes an intelligent vending machine, a mobile terminal, and a server. The automatic settlement system can adopt the automatic shopping method described in the above embodiment. The user recognizes the identification code on the intelligent vending machine through the mobile terminal, and the server establishes the user's shopping event. The cameras with different viewing angles begin to capture shopping videos. The cameras begin to capture shopping videos after the intelligent vending machine door is opened or after the user enters a preset range. When the user leaves the preset shopping range or the intelligent vending machine door is closed, the cameras stop capturing shopping videos and transmit the shopping videos to the server. The server generates the user's order information based on the shopping video and sends it to the mobile terminal. The user performs self-settlement or sets automatic settlement through the order information on the mobile terminal. The automatic settlement system provides users with better self-shopping selectivity and high order accuracy, which can improve the user's shopping experience.

[0203] Example 6

[0204] The present invention provides an intelligent vending machine device and a storage medium, such as Figure 13 As shown, the system includes at least one processor, at least one memory, and computer program instructions stored in the memory.

[0205] Specifically, the above-mentioned processor may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention. The smart vending machine is provided with a cabinet door that can cover the entire merchandise area at the merchandise area position. The cabinet door is a movable cabinet door that can be opened and closed. At the same time, the smart vending machine also includes a camera, a QR code, a bar code and other identification devices that facilitate shopping.

[0206] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside the data processing device. In a specific embodiment, the memory is a non-volatile solid-state memory. In a specific embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0207] The processor reads and executes computer program instructions stored in the memory to implement any one of the multi-perspective commodity recognition and intelligent order generation methods in the first embodiment and the intelligent order generation method based on video segmentation processing based on commodity area weight changes.

[0208] In one example, the electronic device may further include a communication interface and a bus, wherein the processor, the memory, and the communication interface are connected via the bus and communicate with each other.

[0209] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.

[0210] Bus comprises hardware, software or both, couples the parts of electronic equipment to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus can comprise one or more buses.Although the embodiment of the present invention describes and shows specific bus, the present invention considers any suitable bus or interconnection.

[0211] In summary, the embodiments of the present invention provide a method for intelligently generating orders for goods through multi-perspective identification, a method for intelligently generating orders by segmenting videos based on weight changes in product areas, a device, an intelligent vending machine, and a storage medium.

[0212] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0213] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in unit, a function card or the like. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An order generation method based on multi-level commodity detection, characterized in that: The method comprises: Obtain the target video and weight change information of the product area where the user shops through the smart vending machine; According to the weight change information, each frame image of the target video is divided into a first frame image and non-first frame images other than the first frame image according to the acquisition time; Adding a basic confidence level to each target in each frame of the target video according to the first frame image and the non-first frame images; According to the basic confidence of each product and the detection results of each frame image of the target video input into the target detection network, order information corresponding to each target product is obtained; The step of dividing each frame image of the target video into a first frame image and non-first frame images other than the first frame image according to the acquisition time based on the weight change information includes: Segmenting the target video according to each time information of the weight change information to obtain a plurality of target sub-videos; Dividing each frame image of each target sub-video into the first frame image and the non-first frame image according to the acquisition time; The target video is segmented according to each time information of the weight change information to obtain a plurality of target sub-videos, including: Segmenting the target video according to each time information of the weight change information to obtain a plurality of first videos; According to the increase and decrease information of the weight change information, each of the first videos is divided into the target sub-videos corresponding to the product being put on the shelf and the product being taken off the shelf; The step of obtaining order information corresponding to each target product based on the basic confidence of each product and the detection results of each frame image of the target video input into the target detection network includes: Using the target detection network to perform target detection on each frame of the target video to obtain basic product information of each product; According to the confidence of the basic product information of each product, combined with the basic confidence of each product, the product information including the target confidence of each product is obtained; Deduplication of the products according to the product location information of each product information to obtain target products; Outputting order information corresponding to each target product according to the product information of each target product; Deduplication of each product is performed based on the product location information of each product information to obtain each target product, including: Partitioning each frame image of the target video into image regions of each viewing angle to obtain image sub-regions corresponding to the image of each viewing angle; Obtaining, based on the image sub-regions and imaging size information corresponding to the respective products, a relative relationship between the imaging size information of the respective products belonging to the same image sub-region at different viewing angles; The products are deduplicated according to the relative relationship of the imaging size information of the products to obtain the target products.

2. The order generation method based on multi-level commodity detection according to claim 1 is characterized in that: The adding a basic confidence to each target in each frame image of the target video according to the first frame image and the non-first frame images includes: Determining, based on the image information of the first frame image, the positioning information of the commodity area to which each commodity in the first frame image belongs; According to the positioning information, a basic confidence level is added to each target in each frame image of the target video.

3. The order generation method based on multi-level commodity detection according to claim 2 is characterized in that: The method of obtaining the target video and weight change information of the commodity area purchased by the user through the smart vending machine includes: Obtain a status video of the smart vending machine, dividing the product placement area of the smart vending machine into multiple virtual product areas along the shelf arrangement direction of the smart vending machine, and the current status of the smart vending machine; Analyze each frame of the status video to obtain status information of the smart vending machine cabinet door, wherein the status information includes an open state and a closed state; Control each camera to collect video data of the corresponding commodity area according to the on state, and control each camera to stop collecting video data of the corresponding commodity area according to the off state, to obtain each basic video; Physically splicing each frame image of each basic video according to the corresponding frame image of the acquisition time sequence to obtain the target video; The physical stitching means that the stitched image is the sum of the sizes of all images involved in the stitching.

4. An order generation device based on multi-level commodity detection, characterized in that: The device comprises: Video acquisition module: used to obtain target videos of users shopping at smart vending machines and weight change information of product areas; A video segmentation module is configured to divide each frame image of the target video into a plurality of first frame images and a plurality of non-first frame images other than the first frame image according to the acquisition time based on the weight change information; A data processing module is configured to add a basic confidence level to each target in each frame of the target video according to the first frame image and the non-first frame images; An order generation module is configured to obtain order information corresponding to each target product based on the basic confidence level of each product and the detection results of each frame image of the target video input into the target detection network; The step of dividing each frame image of the target video into a first frame image and non-first frame images other than the first frame image according to the acquisition time based on the weight change information includes: Segmenting the target video according to each time information of the weight change information to obtain a plurality of target sub-videos; Dividing each frame image of each target sub-video into the first frame image and the non-first frame image according to the acquisition time; The target video is segmented according to each time information of the weight change information to obtain a plurality of target sub-videos, including: Segmenting the target video according to each time information of the weight change information to obtain a plurality of first videos; According to the increase and decrease information of the weight change information, each of the first videos is divided into the target sub-videos corresponding to the product being put on the shelf and the product being taken off the shelf; The step of obtaining order information corresponding to each target product based on the basic confidence of each product and the detection results of each frame image of the target video input into the target detection network includes: Using the target detection network to perform target detection on each frame of the target video to obtain basic product information of each product; According to the confidence of the basic product information of each product, combined with the basic confidence of each product, the product information including the target confidence of each product is obtained; Deduplication of the products according to the product location information of each product information to obtain target products; Outputting order information corresponding to each target product according to the product information of each target product; Deduplication of each product is performed based on the product location information of each product information to obtain each target product, including: Partitioning each frame image of the target video into image regions of each viewing angle to obtain image sub-regions corresponding to the image of each viewing angle; Obtaining, based on the image sub-regions and imaging size information corresponding to the respective products, a relative relationship between the imaging size information of the respective products belonging to the same image sub-region at different viewing angles; The products are deduplicated according to the relative relationship of the imaging size information of the products to obtain the target products.

5. An intelligent vending machine, characterized in that: include: At least one processor, at least one memory, and computer program instructions stored in the memory, which implement the method according to any one of claims 1 to 3 when the computer program instructions are executed by the processor.

6. A medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images