Retail terminal inventory replenishment method, device and system based on multi-modal data

Abstract

The invention relates to the technical field of intelligent retail terminals, solves the problem of low commodity replenishment accuracy in an unmanned vending cabinet in the prior art, and provides a retail terminal inventory replenishment method, device and system based on multi-modal data. The method comprises the following steps: decomposing real-time video data into multiple frames of real-time images according to the real-time video data of the unmanned vending cabinet in a commodity transaction scene; performing multi-modal feature extraction on the commodity region image to determine commodity feature information; performing human body feature extraction on the user area image in the historical commodity transaction scene, and updating user group feature information; inputting the user group feature information as incremental data into the commodity demand prediction initial model, and determining a commodity demand prediction target model; and inputting the commodity feature information into a commodity demand prediction target model, and screening out a target commodity from a preset commodity list according to a prediction result, and supplementing and putting the target commodity on a shelf. According to the invention, the replenishment accuracy of the unmanned vending cabinet can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent retail terminals, and in particular to a retail terminal inventory replenishment method, device and system based on multimodal data. Background Art

[0002] With the widespread use of vending machines in daily life, how to accurately predict product demand and efficiently replenish goods has become the key to improving operational efficiency. Vending machines not only need to adjust the inventory of goods according to customers' purchasing habits, but also need to monitor goods and customer behavior in real time to achieve intelligent replenishment management. Traditional replenishment methods rely on manual or fixed rules, which leads to problems of excess or insufficient inventory and cannot effectively respond to changing demand. In order to improve the service quality of vending machines, intelligent replenishment systems have gradually gained attention, especially those based on computer vision and deep learning technology, which can combine product characteristics and customer behavior to predict demand and dynamically replenish goods.

[0003] Existing technologies mainly rely on commodity demand forecasting methods based on time series data and historical sales records. These methods have poor adaptability in dynamic environments and cannot reflect changes in customer behavior in real time. In addition, although there are some commodity detection solutions based on visual recognition technology, most of them focus on recognition in static images or preset scenes, and it is difficult to handle complex transaction scenarios in vending machines in real time.

[0004] The existing Chinese patent CN114219412A discloses an automatic replenishment method based on sales forecast of a smart commodity system, including: obtaining the inventory quantity, inventory cycle and commodity attributes of each of a plurality of inventory commodities; judging whether there is a commodity that needs to be replenished and has insufficient inventory among the plurality of inventory commodities, and if so, retrieving the historical sales database of the smart commodity system, and performing the following steps for each commodity that needs to be replenished to calculate the replenishment quantity: screening out commodities whose matching degree with the commodity that needs to be replenished in commodity attributes exceeds a preset degree from the retrieved historical sales database as sample commodities; obtaining sales information of the sample commodities, identifying sales change data of the sales information, comparing the sales trends of the commodity that needs to be replenished and the sample commodities, and calculating according to the comparison result of the sales trends of the two: the sales quantity of the sample commodities within the remaining sales time of the current inventory cycle of the commodity that needs to be replenished; using the calculated sales quantity of the sample commodities as the replenishment quantity of the commodity that needs to be replenished; and allocating inventory commodities according to the calculated replenishment quantities of each commodity that needs to be replenished to achieve replenishment. Although the above scheme predicts the replenishment quantity by matching the historical sales database with product attributes, it faces the following technical problems in actual application: first, the matching degree of product attributes is not completely accurate, resulting in the selection of sample products being unrepresentative, thus affecting the calculation of the replenishment quantity; second, the comparison of sales trends is affected by external factors (such as seasonal fluctuations, promotional activities, etc.), resulting in inaccurate sales trend analysis, which in turn affects the accuracy of replenishment prediction; in addition, the time synchronization problem of inventory cycle and sales changes also leads to the risk of lagging or over-replenishment in the calculated replenishment quantity.

[0005] Therefore, how to improve the accuracy of replenishing goods in unmanned vending machines is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of this, the present invention provides a retail terminal inventory replenishment method, device and system based on multimodal data, so as to solve the problem of low accuracy of replenishment of goods in unmanned vending machines in the prior art.

[0007] The technical solution adopted by the present invention is: In a first aspect, the present invention provides a retail terminal inventory replenishment method based on multimodal data, the method comprising: According to the real-time video data of the unmanned vending machine in the commodity transaction scenario, the real-time video data is decomposed into multiple frames of real-time images; Perform commodity target detection on each frame of the real-time image, and determine a commodity area image according to the target detection result; Performing multimodal feature extraction on the product area image to determine product feature information; Extract human features from user area images in historical commodity transaction scenarios and update user group feature information; Inputting the user group characteristic information as incremental data into a pre-trained commodity demand forecasting initial model for fine-tuning training to obtain a fine-tuned commodity demand forecasting target model; The commodity feature information is input into the commodity demand forecasting target model, and target commodities are selected from a preset commodity list for replenishment and listing according to the forecasting result.

[0008] In a second aspect, the present invention provides a retail terminal inventory replenishment device based on multimodal data, the device comprising: An image acquisition module, used to decompose the real-time video data of the unmanned vending machine in the commodity transaction scenario into multiple frames of real-time images; The target detection module is used to perform commodity target detection on each frame of the real-time image, and determine the commodity area image according to the target detection result; A multimodal feature extraction module, used to extract multimodal features from the commodity area image to determine commodity feature information; A human feature extraction module is used to extract human features from user area images in historical commodity transaction scenarios and update user group feature information; A target model training module, used for inputting the user group characteristic information as incremental data into a pre-trained commodity demand forecasting initial model for fine-tuning training, and obtaining a fine-tuned commodity demand forecasting target model; The replenishment and shelf-putting module is used to input the commodity feature information into the commodity demand prediction target model, and select the target commodity for replenishment and shelf-putting from the preset commodity list according to the prediction result.

[0009] In the third aspect, an embodiment of the present invention also provides a retail terminal inventory replenishment system based on multimodal data, including: a first camera and a second camera; the collection range of the first camera is inside and outside the unmanned vending machine, and the collection range of the second camera is outside the unmanned vending machine; at least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, the method as described above is implemented.

[0010] In summary, the beneficial effects of the present invention are as follows: The present invention provides a retail terminal inventory replenishment method, device and system based on multimodal data, the method comprising: decomposing the real-time video data of an unmanned vending machine in a commodity transaction scenario into multiple frames of real-time images; performing commodity target detection on each frame of the real-time image, and determining the commodity area image according to the target detection result; performing multimodal feature extraction on the commodity area image to determine commodity feature information; performing human body feature extraction on the user area image in the historical commodity transaction scenario, and updating the user group feature information; inputting the user group feature information as incremental data into the pre-trained commodity demand prediction initial model for fine-tuning training to obtain a fine-tuned commodity demand prediction target model; inputting the commodity feature information into the commodity demand prediction target model, and selecting the target commodity from the preset commodity list for replenishment and shelf according to the prediction result. The present invention can reflect changes in customer behavior in real time, avoid the problem of delayed or over-replenishment of commodity replenishment, and thus improve the accuracy of replenishment in unmanned vending machines. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0012] Figure 1 This is a schematic diagram of the overall working process of the retail terminal inventory replenishment method based on multimodal data in Example 1 of the present invention; Figure 2 This is a schematic diagram of a process of performing commodity target detection on each frame of the real-time image in Embodiment 1 of the present invention; Figure 3 It is a schematic diagram of the process of extracting multimodal features from the commodity area image in Embodiment 1 of the present invention; Figure 4 It is a schematic diagram of a process of performing multimodal fusion on the color feature information, contour feature information, texture feature information and the text feature information in Embodiment 1 of the present invention; Figure 5 This is a flow chart of extracting human features from a user area image in a previous commodity transaction scenario in Embodiment 1 of the present invention; Figure 6 This is a flow chart of inputting the user group characteristic information as incremental data into the pre-trained commodity demand forecasting initial model for fine-tuning training in Embodiment 1 of the present invention; Figure 7 This is a schematic diagram of a process of selecting target commodities for replenishment and shelf placement from a preset commodity list according to prediction results in Embodiment 1 of the present invention; Figure 8 This is a schematic diagram of a process of selecting target commodities for replenishment and shelf placement from a preset commodity list according to an optimization processing result in Embodiment 1 of the present invention; Fig. 9 It is a structural block diagram of a retail terminal inventory replenishment device based on multimodal data in Embodiment 3 of the present invention; Fig.10 This is a schematic diagram of the structure of the smart retail terminal in Embodiment 4 of the present invention; Fig.11 is a schematic diagram of the structure of an electronic device in Embodiment 4 of the present invention; The reference numerals are as follows: 1-First camera; 2-Second camera. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the phrase "comprising..." do not exclude the existence of other identical elements in the process, method, article or device comprising the elements. If there is no conflict, the embodiments of the present invention and the various features in the embodiments can be combined with each other, all within the protection scope of the present invention.

[0014] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant location, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0015] Example 1

[0016] See also Figure 1 Embodiment 1 of the present invention discloses a retail terminal inventory replenishment method based on multimodal data, the method comprising: According to the real-time video data of the unmanned vending machine in the commodity transaction scenario, the real-time video data is decomposed into multiple frames of real-time images; Specifically, the real-time video data of the vending machine is obtained, the transaction scene in the vending machine is captured in real time, and the real-time video data is decomposed into multiple frames, that is, each second or less of the video stream is divided into independent image frames. The camera is used to continuously record the commodity transaction process, and then the video data is converted into a series of continuous image frames through video processing technology (such as frame extraction algorithm) so that each frame can be subsequently processed. The real-time image can accurately capture the status of the commodity, customer behavior and transaction dynamics at each time point, providing timely and accurate visual data support for subsequent commodity detection and demand forecasting, ensuring that the replenishment system can make real-time decisions based on the latest situation, rather than relying on delayed or outdated data.

[0017] Perform commodity target detection on each frame of the real-time image, and determine a commodity area image according to the target detection result; Specifically, the commodity target detection technology is applied to each frame of real-time image to identify the commodity in the image and determine its area. A deep learning model, such as YOLO and Faster R-CNN, is used to perform target detection on each frame of the real-time image. After training, the model can identify the commodity in the image and extract the location information of each commodity to generate a commodity area image. The commodity area image refers to the image part containing the commodity, removing other irrelevant backgrounds or objects. Target detection identifies the commodity by analyzing the features in the image (such as shape, color, texture, etc.) and selects the commodity area. Therefore, it can accurately extract commodity information from the complex vending machine environment, laying the foundation for subsequent feature extraction and demand forecasting. By accurately identifying the commodity area, focusing on the commodity itself and avoiding interference from other background factors, the accuracy and real-time performance of commodity monitoring are improved, ensuring more accurate replenishment decisions.

[0018] In one embodiment, see Figure 2, performing commodity target detection on each frame of the real-time image, and determining the commodity area image according to the target detection result includes: Resize and denoise each frame of the real-time image to determine multiple frames of target images after denoising; Specifically, the real-time image is first resized to ensure that the image meets the input requirements of the target detection model (such as uniform size). Then, the noise in the image is removed through denoising processing (such as Gaussian filtering, mean filtering and other technologies) to reduce image interference caused by ambient light, camera quality or other factors. The denoised image can more clearly display the characteristics of goods and users, and improve the accuracy of target detection. Because denoising processing can remove interference information, subsequent target detection can focus more on the real target object and avoid misidentification and missed detection.

[0019] Inputting the target image of each frame into a pre-trained target detection model to determine initial target boundary position information, wherein the initial target includes a product target and a user target; Specifically, the denoised target images of each frame are input into a pre-trained target detection model (such as YOLO, SSD, etc.). The target detection model is trained with a large amount of labeled data and can accurately identify the goods and user targets in the image; the target detection model outputs the bounding box (position and size) of each product target, that is, the initial boundary position information of the product target; the initial targets include goods and users, so the model can distinguish these two types of targets and give their location information. The beneficial effect of this step is to provide accurate target positioning information for subsequent processing, ensuring that goods and users can be accurately distinguished and subsequently analyzed.

[0020] Acquire an initial target area image within the initial target boundary according to the initial target boundary position information; Specifically, based on the initial target bounding box detected, the image is cropped to obtain the area within the bounding box, and the initial target area image containing the product and the user is obtained; each target area is extracted separately for subsequent feature processing. Through accurate initial target area extraction, the interference of irrelevant information is reduced, making subsequent image processing more focused and able to more effectively analyze the details of the product and the user.

[0021] The commodity label detection is performed on the initial target area image, and the commodity area image is determined according to the detected commodity label position information and the initial target boundary position information.

[0022] Specifically, the initial target area image is further tested for product labels. The main purpose is to identify whether there are labels on the product, such as barcodes and price tags, and determine the location information of the labels. By combining the initial target boundary location information, the product labels are accurately extracted and matched with the product target area to determine the final product area image. Label detection can accurately locate the specific information of the product (such as price, type, etc.), further ensure the accurate division of the product area, and provide a more accurate basis for subsequent product analysis and demand forecasting.

[0023] Performing multimodal feature extraction on the product area image to determine product feature information; Specifically, multimodal feature extraction is performed on the product area image, with the purpose of extracting multiple types of feature information from the product area image to comprehensively describe the characteristics of the product. Multimodal feature extraction includes using a deep learning model, such as a convolutional neural network (CNN), to extract visual features of the image, such as color, shape, and texture, while combining other modal information, such as data such as product weight and temperature collected by embedded sensors, or customer interaction information captured by voice recognition. These multiple types of features are fused together to form comprehensive feature information of the product. This multimodal processing can make full use of the advantages of different data sources and provide a more comprehensive and accurate description of the product than a single visual feature, thereby better supporting product demand forecasting, inventory management, and user behavior analysis. Through multimodal feature extraction, the various characteristics of the product can be accurately captured, and the accuracy and intelligence level of replenishment forecasting can be improved.

[0024] In one embodiment, see Figure 3 , the performing multimodal feature extraction on the commodity area image to determine commodity feature information includes: Performing color space conversion on the commodity area image, and obtaining a color histogram of the commodity area in the converted color space; Specifically, the product area image is first converted into a color space, converting the original image from the RGB color space to other color spaces that are more suitable for feature extraction, such as HSV or Lab. Color space conversion can better highlight the color information related to the product in the image and reduce the impact of changes in ambient light. Then, based on the converted color space, the color histogram of the product area is calculated. The color histogram reflects the distribution of different colors in the image and can be used to distinguish the color characteristics of the product. The beneficial effect of this step is that through accurate color feature extraction, the main color of the product can be identified, which helps to distinguish different products and improves the reliability and accuracy of image analysis.

[0025] Aggregating and optimizing the color histogram to determine color feature information corresponding to the product area; Specifically, the color histogram generated in the previous step is aggregated and optimized. Aggregation includes merging similar color areas to reduce noise; optimization includes removing insignificant color information and enhancing the representative color features of the product. The purpose of this process is to extract more stable and representative color features and avoid color deviations caused by factors such as changes in ambient light. The color feature information corresponding to the product area finally obtained will more accurately describe the color characteristics of the product, which can improve the stability and recognition of the color features, so that the product can still be accurately identified in different environments.

[0026] Performing edge detection on the commodity area image, and determining contour feature information of the commodity area according to the edge detection result; Specifically, edge detection algorithms, such as Canny edge detection, are applied to product area images to extract the contour features of the products. Edge detection identifies places in the image where grayscale changes significantly, marks the boundaries of objects, and then outlines the shape of the products. By analyzing the contours, the geometric shape features of the products, such as rectangles and circles, are extracted. This can effectively distinguish the appearance features of the products, especially for products with large shape differences, which helps to improve the accuracy and robustness of recognition.

[0027] Extracting texture features from the commodity area image to determine texture feature information of the commodity area; Specifically, Local Binary Pattern (LBP) is a technology widely used in image texture analysis. By performing LBP conversion on the product area image, detailed texture features of the product surface, such as roughness and smoothness, are extracted. By comparing the grayscale value of the pixel with the surrounding pixels, the image is converted into a binary pattern to capture local texture features. This can distinguish the surface textures of different products, for example, whether the product has a smooth or rough surface. It can effectively process texture information and has strong noise resistance. It can identify the unique texture features of products in complex environments, thereby improving the accuracy of product recognition.

[0028] Performing text recognition on the commodity area image to determine text information in the commodity area; Specifically, optical character recognition (OCR) technology is used to identify text in product area images and extract labels, brand names, prices or other information on the products. OCR technology analyzes the text area in the product area image, identifies the characters in the product area and converts them into machine-readable text information. It can accurately extract the text information of the product and provide more data support for subsequent product identification and demand forecasting. In particular, for products with logos and labels, it can quickly obtain important product information.

[0029] Performing natural language processing on the text information to determine text feature information; Specifically, once the text information is recognized through OCR, natural language processing (NLP) technology is applied to further process the text, including word segmentation, part-of-speech tagging, entity recognition and other operations to extract key information in the product description, such as brand, model, type, and price. The text feature information obtained after NLP processing is used for deeper analysis, such as product classification and sales trend prediction. It can convert the text information of the product into structured features to support more intelligent data analysis and decision-making.

[0030] Multimodal data fusion is performed on the color feature information, the contour feature information, the texture feature information and the text feature information to determine the commodity feature information.

[0031] Specifically, the extracted color feature information, contour feature information, texture feature information and text feature information are multimodally fused to generate comprehensive feature information of the product. Multimodal fusion methods include weighted averaging, splicing or deep learning models (such as multimodal neural networks) to combine information from these different sources to obtain a more comprehensive and accurate description of the product. By fusing visual features and text features, the identification of the product does not only rely on a single data source, but also comprehensively considers the various features of the product, greatly improving the accuracy and robustness of the identification.

[0032] In one embodiment, see Figure 4 , the performing multimodal data fusion on the color feature information, the contour feature information, the texture feature information and the text feature information to determine the commodity feature information comprises: According to the contour feature information, the product area image is segmented to obtain the local area position information corresponding to the product body; Specifically, based on the extracted product contour feature information, image segmentation technology, such as edge-based segmentation algorithm or region growing algorithm, is used to extract the product area from the background, identify the specific area of ​​the product body, and accurately separate the product from the image by analyzing the contour, and provide a clear target area for subsequent feature extraction. By accurately determining the boundary of the product body, accurate local area images are provided for subsequent feature extraction and classification, thereby improving the accuracy and efficiency of target recognition.

[0033] Extracting local area color feature information from the color feature information according to the local area position information; Extracting local area contour feature information from the contour feature information according to the local area position information; Extracting local area texture feature information from the texture feature information according to the local area position information; Extracting local area text feature information from the text feature information according to the local area position information; Specifically, based on the local area image obtained by segmentation, the local area color feature information, local area contour feature information, local area texture feature information and local area text feature information of the local area are further extracted, which can more comprehensively describe the visual and textual attributes of the product area and provide more powerful data support for subsequent product identification and classification.

[0034] Inputting the commodity area image into a pre-trained target classification model to determine the commodity category; Specifically, the extracted product area image is input into a pre-trained target classification model, such as a deep convolutional neural network (CNN), which has been trained with a large number of labeled product images and can identify and classify the types of products in the image. The output is the category to which the product belongs, such as beverages, snacks, daily necessities, etc. This process can automatically and accurately determine the category of the product, improve the automation of product identification, and reduce the risk of manual intervention and misclassification.

[0035] Determine, according to the commodity category and the mapping relationship table between the preset commodity category and the weight, a first weight of the local area color feature information, a second weight of the local area contour feature information, a third weight of the local area texture feature information, and a fourth weight of the local area text feature information; Specifically, according to the commodity category, the preset mapping relationship table between the commodity category and the weight is consulted, and different weights are assigned to different features of the local area of ​​different categories of commodities, such as color, outline, texture, and text, to determine the first weight of the local area color feature information, the second weight of the local area outline feature information, the third weight of the local area texture feature information, and the fourth weight of the local area text feature information; the mapping table adjusts the relevance of each feature according to different commodity types. For example, for some commodities, the color feature may be more important than the texture feature, and vice versa. The weight of each feature in commodity identification is flexibly adjusted according to the actual characteristics of the commodity, which improves the accuracy of multi-feature fusion and makes commodity classification and demand prediction more accurate.

[0036] According to the first weight, the second weight, the third weight and the fourth weight, the local area color feature information, the local area contour feature information, the local area texture feature information and the local area text feature information are weightedly fused, and the fused feature information is determined as the product feature information.

[0037] Specifically, according to the first weight, the second weight, the third weight and the fourth weight previously determined, the weighted averaging method is used to fuse the local area color feature information, the local area contour feature information, the local area texture feature information and the local area text feature information. After weighted fusion, a fused commodity feature information is obtained. The commodity feature information contains a complete description of the commodity in each dimension. Through this weighted fusion, the importance of each feature can be effectively combined to ensure the reasonable integration of feature information. Through weighted fusion, a more accurate and comprehensive commodity feature description can be obtained, thereby improving the intelligence and accuracy of commodity identification, demand forecasting and replenishment management.

[0038] Extract human features from user area images in historical commodity transaction scenarios and update user group feature information; Specifically, feature information related to the human body is extracted from user area images captured in historical commodity transaction scenarios. This process usually includes the use of deep learning models, such as human posture estimation, to analyze the human body structure and behavioral characteristics in the image, such as the user's posture, movement, height, body shape, and movement trajectory. Through human feature extraction, the user's physical characteristics are identified and analyzed, and the customer's behavioral changes are understood in real time, thereby more accurately predicting commodity demand, further optimizing replenishment decisions and the service experience of vending machines. Through this extraction of user group feature information, unmanned vending machines can respond to customer needs more intelligently, improve operational efficiency and customer satisfaction.

[0039] In one embodiment, see Figure 5 The extracting of human features from user area images in historical commodity transaction scenarios and updating of user group feature information include: Obtain historical images of commodity trading scenarios; Specifically, historical images of commodity transaction scenarios are retrieved from historical order data. Historical images are usually captured in real time by cameras or other monitoring equipment of unmanned vending machines during the interaction between customers and unmanned vending machines. The purpose of obtaining historical images is to capture and record customer behavior and interaction information. These data will serve as the basis for subsequent analysis.

[0040] Performing human body key point detection on the historical image to determine the user area image; Specifically, human key point detection algorithms, such as OpenPose or HRNet, are used to process historical images and detect human key points in the images, such as the head, shoulders, elbows, knees, etc. The human key point position information can help accurately identify and locate the customer's position and posture in the image. Through the detected human key point information, a sub-image containing the customer area, namely the user area image, can be extracted from the entire image, thereby accurately identifying the customer's position and activities in front of the vending machine, making subsequent feature extraction and behavior analysis more accurate.

[0041] Performing image enhancement processing on the user area image to determine an enhanced user target image, wherein the image enhancement processing includes super-resolution reconstruction, image denoising and contrast adjustment; Specifically, in this step, the image extracted from the user area is enhanced, and the resolution of the image is improved through the super-resolution algorithm, so that the details in the user area are clearer, which is convenient for further feature analysis; the noise in the image is removed by using the denoising technology to reduce the image quality degradation caused by shooting conditions (such as low light or interference factors); the image contrast is optimized to make the target area in the image (such as the customer's physical features) more prominent, which is convenient for subsequent feature extraction and analysis. By improving the quality of the user area image, clearer and more accurate visual information is provided for subsequent feature extraction, and the system's perception of customer behavior is enhanced.

[0042] Inputting the user target image into a pre-trained posture detection model to determine posture feature information; Specifically, the user target image is first input into a pre-trained posture detection model. The posture detection model is based on a deep learning framework, such as OpenPose, HRNet, and MoveNet, and analyzes the positions of key human body points in the user target image, thereby identifying human body posture information, including standing, walking, raising arms, bending over, and other actions. By identifying these actions, the customer's behavior state and intention can be understood, such as whether the customer is selecting goods, picking up goods, or paying. Through the posture detection model, the user's posture feature information can be obtained in the user target image, which not only helps to improve the accurate identification of the user, but also provides more intelligent services, such as helping replenishment personnel optimize replenishment strategies, understand customers' purchasing decisions, etc. Combining these features can help replenishment personnel respond dynamically according to customer behavior, improve customer experience, and optimize operational efficiency.

[0043] Performing clothing type recognition on the user target image to determine clothing feature information of the user; Specifically, the clothing type recognition is performed on the user target image. The goal of clothing type recognition is to accurately identify the type, color, and style of clothing worn by the user. Specifically, a deep learning model such as a convolutional neural network (CNN) is used to extract features of clothing areas. The clothing area feature information includes but is not limited to clothing styles, such as T-shirts, shirts, pants, jackets, colors, such as red, blue, black, patterns, such as stripes, solid colors, and prints, and materials, such as cotton, synthetic fibers, and wool. By extracting these detailed features of clothing, the user's wearing style is further analyzed. In order to ensure accurate recognition of clothing types, a pre-trained model trained with a large-scale data set is relied on. The data set contains various types of clothing images, covering different human body postures, lighting conditions, and viewing angles. The pre-trained model can cope with the diversity in real scenes and can recognize the user's clothing in complex environments. In addition, the extracted clothing features will be further processed, such as by color space conversion, such as RGB to HSV, to enhance the stability of color recognition, or use edge detection technology to improve the recognition accuracy of clothing contours. Through this process, we can not only know what type of clothing the user is wearing, but also infer the user's activity scene or needs. For example, wearing sportswear may mean that the user is exercising, and wearing formal clothes may mean that the user is working. Through these inferences, personalized product replenishment recommendations can be made to replenishment personnel based on the user's clothing characteristics. For example, if the user is wearing sportswear, it is inferred that he or she needs to buy sports drinks; this refined analysis can improve the user experience and enhance the personalization of services, thereby improving the operating efficiency and user satisfaction of vending machines.

[0044] Preliminarily predicting the age stage and gender of the user based on the clothing feature information and the user posture feature information, and determining an initial age range and user gender; Specifically, based on the clothing feature information and posture feature information extracted from the user's image, a multimodal learning model is used to make a preliminary prediction of the user's age and gender. Specifically, the user's dressing style is first judged through information such as clothing type, color, and style, which provides preliminary clues for the prediction of age and gender. For example, certain clothing styles or colors may be associated with specific age groups or gender groups, such as sportswear or casual wear that are usually associated with young people, while formal suits are more common on middle-aged and elderly men. Secondly, the user's posture features (such as standing, sitting, walking, etc.) can also provide important behavioral patterns. Clues, posture features are related to typical behavior patterns of age or gender. Combining these multiple features, a preliminary prediction of the user's age stage, such as children, teenagers, adults, the elderly and gender (male or female) is made through machine learning or deep learning models. Pre-trained classification models, such as support vector machines (SVM), decision trees, and deep neural networks are used to comprehensively analyze these multi-dimensional features. The prediction results will give an initial age range (such as 18-25 years old, 26-35 years old) and gender (male or female), providing basic data for subsequent personalized product replenishment recommendations and service adjustments to replenishment personnel.

[0045] Performing face detection on the user target image to determine whether there is a user face in the user target image; Specifically, the user target image is processed through face detection algorithms, such as Haar Cascade, MTCNN, and Dlib, to determine whether there is a recognizable user facial area in the image. Facial detection technology determines the position of the face and whether it is in the image by identifying facial features, such as the relative position and geometric relationship of the eyes, nose, and mouth. If there are no clear facial features in the image, this step will return a result that the face does not exist, and the subsequent facial feature extraction process will be skipped. If a face is detected, the subsequent steps will continue. By detecting whether there is a face in the image, the system can ensure that only when the user's facial information is clear, it will enter a more detailed feature extraction process, which effectively avoids misoperation or invalid calculations and improves overall processing efficiency and accuracy.

[0046] If the user's face exists, facial feature extraction is performed on the facial region image to determine facial feature information, wherein the facial feature information includes eye feature information and facial texture feature information; Specifically, after the face is detected, accurate feature extraction is performed on the facial region image. Facial feature extraction usually uses deep learning models such as CNN, FaceNet or classic computer vision algorithms such as HOG features and LBP texture analysis to identify key facial areas. First, eye feature information is extracted by detecting the position, shape, size and other features of the eyes. This information is closely related to age and gender. Secondly, facial texture feature information is obtained by analyzing the texture, wrinkles, details and other aspects of the skin surface. This is particularly useful for age judgment. The extracted eye and texture features will form a set of vector information containing facial features. By extracting detailed facial features, more effective information that helps to infer age and gender can be obtained. Accurate extraction of facial feature information makes age judgment more accurate, especially when the age range is vague. The details of the eye and facial texture can be effectively used as a basis for correction, thereby improving the accuracy of the prediction.

[0047] According to the eye feature information and the facial texture feature information, the initial age range is corrected for a second time, and the corrected age range is determined as the target age range; Specifically, according to the eye feature information and the facial texture feature information, by analyzing the eye features and texture features of the facial area, the initial age range of the preliminary prediction is secondary corrected. Eye feature information, such as eye bags and wrinkles at the corners of the eyes, and facial texture feature information, such as skin laxity and the appearance of fine lines, can provide more accurate age estimation. Specifically, facial aging features (such as wrinkles at the corners of the eyes and the degree of skin laxity) can effectively help distinguish subtle differences in age. Through the deep learning model, the initial age range is fine-tuned in combination with the eye feature information and the facial texture feature information to ensure that the judgment of the age group is more accurate. Through the secondary correction of facial features, the preliminary prediction results can be refined, especially in the absence of obvious age markers (such as skin color or clothing), facial details provide accurate age judgment, further improving the accuracy.

[0048] According to the geographical location information of the unmanned vending machine and the clothing feature information, the occupation of the user is predicted and determined; Specifically, the geographical location information and clothing feature information of the unmanned vending machine are used to predict the user's occupation; first, the geographical location information of the unmanned vending machine is obtained through GPS positioning, and the use scene of the unmanned vending machine is determined, such as a business district, office area, industrial area or campus scene, and the main consumer group characteristics of the location are analyzed in combination with historical transaction data; then, the clothing feature information is obtained and analyzed, for example, white coats correspond to the medical industry, and school uniforms correspond to students; according to the geographical location information and clothing feature information, the user's occupation is predicted using a classification method. For example, the user's occupation is predicted using a decision tree classification method. First, the user is divided into different scene categories, such as office area, industrial area, campus or hospital, using the geographical location information; then, it is further subdivided based on the clothing feature information, for example, if the geographical location is in the hospital and the user is identified as a white coat according to the clothing feature, then the user's occupation is a doctor or nurse; if the geographical location is in the school and the user is identified as a school uniform according to the clothing feature, then the user's occupation is a student; the training data can be continuously optimized through historical transaction records and known user occupation data, and finally a stable classification model is formed to achieve efficient occupation prediction, optimize the occupation prediction model, and improve the accuracy of user portraits.

[0049] The user group characteristic information is determined according to the user gender, target age range and the user occupation.

[0050] Specifically, after completing the precise correction of the initial age range, the user gender, the user occupation and the corrected target age range are combined to finally determine the user group characteristic information. By combining the characteristics of the three dimensions of gender, occupation and age, a more comprehensive and accurate user portrait is provided, and the user's age group, gender and occupation are determined more accurately, which provides strong support for the subsequent recommendation of product replenishment to the replenishment personnel.

[0051] Inputting the user group characteristic information as incremental data into a pre-trained commodity demand forecasting initial model for fine-tuning training to obtain a fine-tuned commodity demand forecasting target model; Specifically, the user group feature information is input as incremental data into the pre-trained commodity demand forecasting initial model for training. The commodity demand forecasting initial model will extract new information from the user's age, gender, skin color and other features, and use these features as incremental data, combined with the original commodity demand forecasting model, update the model weights, and thus generate a more accurate commodity demand forecasting target model. Specifically, the incremental learning method can seamlessly introduce new data into the existing model without the need to retrain the entire model, which can effectively improve the model's adaptability and predictive ability when real-time data is updated. In this way, the forecast of commodity demand can be dynamically adjusted to improve the accuracy of the forecast.

[0052] In one embodiment, see Figure 6 , the user group characteristic information is input as incremental data into the pre-trained commodity demand forecasting initial model for fine-tuning training, and the fine-tuned commodity demand forecasting target model is obtained, including: Using the pre-collected historical commodity sales data and historical commodity feature information corresponding to each commodity, the long-short term memory model is trained to determine the initial commodity demand forecasting model; Specifically, firstly, the historical commodity sales data and historical commodity feature information are collected and sorted, the historical sales data includes the sales quantity, price, and sales time of the commodity, and the historical commodity feature information includes the type, brand, and specification of each commodity; the historical commodity sales data and historical commodity feature information are preprocessed, including removing outliers, filling missing values, and normalization to ensure the quality of input data. Then, based on the processed historical commodity sales data and historical commodity feature information, the long short-term memory network (LSTM) is trained. Since the LSTM model is good at processing time series data and can capture the long-term dependency and short-term fluctuation of commodity demand changes, it is suitable for commodity demand forecasting. By taking the historical sales data and historical commodity feature information as input, the LSTM model can learn the time series pattern of commodity sales, and gradually adjust the parameters of the model by optimizing the loss function in the training process, such as the mean square error, MSE, to finally obtain an initial model that can predict future commodity demand. After the training process is completed, the validation set is evaluated to ensure that the trained model can better predict the demand for commodities and provide an accurate initial prediction benchmark for subsequent incremental training.

[0053] Classifying the target age range to determine age characteristic information corresponding to the target age range; Specifically, first, the target age range of the extracted users is divided into multiple discrete category ranges, such as the target age range is divided into young people, middle-aged people or elderly people. Each discrete category range is customized according to market demand or product characteristics to ensure that user groups of different age groups can be segmented; through this classification, the model can better identify the demand characteristics and consumption preferences of groups of different age groups when purchasing goods; in order to ensure that the processed features can be input into the model, these category ranges are encoded. A common method is to use one-hot encoding to convert each age range into a binary feature vector. For example, the age range "18-25 years old" may correspond to a vector [1, 0, 0], while "26-35 years old" corresponds to [0, 1, 0], and so on. This encoding method allows the model to make full use of information from different age groups and avoid the problem of sequential dependency between category information, thereby improving the ability of the commodity demand forecasting model to predict the needs of users of different age groups.

[0054] Classifying and encoding the user's occupation to determine occupational characteristic information; Specifically, the user's occupation is classified and feature encoded, with the aim of converting the user's occupation information into numerical features that can be recognized by the model so that it can be used in subsequent data analysis. In terms of specific implementation, it is first necessary to build an occupational category system, such as dividing it into specific occupational categories such as doctors and students. Then, the user's occupation is classified and predicted to ensure that each user corresponds to an occupational category. After completing the occupational classification, the feature encoding method is used to convert the occupational category into a numerical representation. One-hot encoding can be used to assign a binary vector to each occupation, map the user's occupation to a low-dimensional vector space, and determine the occupational feature information, which significantly improves the usability of the user's occupation.

[0055] Converting and encoding the user's gender to determine binary feature information after processing; Specifically, the user's gender information is converted and encoded, and the user's gender is converted into a numerical feature that is directly input into the machine learning model. Since gender is a binary feature, it can usually be represented by binary encoding (also known as one-hot encoding). In binary encoding, the gender feature is converted into two possible numerical labels: one for "male" and the other for "female". In order to adapt to most machine learning algorithms, these gender information is usually converted into binary variables of 0 and 1. For example, suppose "male" is represented by 1 and "female" is represented by 0, or vice versa. With this encoding, the model can directly use these numerical features for training and prediction. In addition, if the gender ratio of the data set is extremely unbalanced, additional processing may be required, such as class balancing techniques (such as oversampling or undersampling) to avoid the model being biased towards a certain gender. Through this processing method, gender information can be converted into numerical features, which can effectively help the prediction model understand the differences in product demand among users of different genders. For some specific scenarios, it can also be expanded to more categories (such as multivariate gender encoding) to support more complex demand prediction.

[0056] According to preset weights, weighted fusion is performed on the age feature information, the occupation feature information and the binary feature information to determine multi-dimensional feature information; Specifically, in the process of weighted fusion, the age feature information, occupational feature information and the binary feature information from different sources are first standardized to ensure their comparability on a numerical scale; the age feature information, occupational feature information and the binary feature information have all been classified and feature-coded, wherein the age feature information is encoded using one-hot coding, the gender feature information is encoded using binary coding, and the occupational feature information is represented using one-hot coding or a low-dimensional embedding vector; during fusion, since the gender feature information has a strong degree of distinction and can directly affect the user's consumption behavior and product preferences, it is given the highest first weight in weight distribution; secondly, the age feature information has a greater impact on user needs. The influence is second only to gender, but it is still an important factor in commodity demand prediction. For example, users of different age groups have significant differences when purchasing electronic products, daily necessities or luxury goods, so it is given the second highest weight; finally, although occupational characteristic information can reflect the user's economic ability, consumption habits and demand scenarios, due to the complexity and diversity of occupational classification, its impact on commodity demand is relatively indirect, so it is given a relatively low third weight; the specific weighted fusion process uses a weighted summation method to calculate the comprehensive feature vector of each user. After weighted calculation, the comprehensive feature vector is obtained as the final multi-dimensional user feature information to achieve accurate demand prediction for different user groups. This fusion method can ensure that the core characteristics of users are reasonably expressed in different consumption scenarios, thereby improving the prediction accuracy of the model.

[0057] The multi-dimensional feature information is input into the commodity demand forecasting initial model for incremental training and model fine-tuning to determine the commodity demand forecasting target model.

[0058] Specifically, the multi-dimensional feature information is input into the initial model for commodity demand prediction, and incremental training and model fine-tuning are performed. The goal of incremental training is to enable the initial model to be further optimized according to the newly added user features (such as age, skin color, gender, etc.) without retraining the entire model. This process can be achieved through methods such as mini-batch gradient descent to ensure that the model can effectively absorb new feature information during training while maintaining memory of historical data. During incremental training, the model gradually adapts to the training data containing new features by updating weights and parameters, thereby improving prediction accuracy. The role of fine-tuning is to make detailed adjustments to the model parameters to ensure that it can still maintain efficient prediction capabilities under the newly added feature dimensions. Finally, the model that has undergone incremental training and fine-tuning is the commodity demand prediction target model, which can more accurately predict commodity demand based on the personalized features of different users (such as age, skin color and gender).

[0059] The commodity feature information is input into the commodity demand forecasting target model, and target commodities are selected from a preset commodity list for replenishment and listing according to the forecasting result.

[0060] Specifically, the commodity feature information is input into the commodity demand prediction target model, and the commodity demand prediction target model trained with incremental data is used for real-time prediction. Based on the accurate prediction of commodity demand by the target model, the most suitable commodities in the preset commodity list are automatically selected for replenishment and listing. In this way, it is possible to intelligently determine which commodities are in high demand in the current transaction scenario, thereby optimizing replenishment decisions, reducing manual intervention and inventory backlogs, and improving the commodity circulation efficiency of the vending machine.

[0061] In one embodiment, see Figure 7 The step of inputting the commodity feature information into the commodity demand forecasting target model and selecting target commodities for replenishment and shelf placement from a preset commodity list according to the forecasting result comprises: Inputting the commodity feature information into the commodity demand forecasting target model, and outputting the commodity target demand; Specifically, the product feature information (such as color, label, shape, texture, etc.) extracted from the unmanned vending machine is input into the trained product demand prediction target model. The target model calculates the demand for each product in the current situation based on the product feature information. This prediction result provides basic data for subsequent replenishment decisions.

[0062] In one embodiment, inputting the commodity feature information into the commodity demand forecasting target model to output the commodity target demand includes: Inputting the commodity characteristic information into the commodity demand forecasting target model to determine the initial demand for the commodity; According to the user group characteristic information and preset commodity replenishment decision parameters, the initial demand for the commodity is corrected to determine the target demand for the commodity, wherein the commodity replenishment decision parameters include sales volume, repurchase rate, gross profit margin, dynamic sales rate and new product rate; Specifically, after determining the initial demand for goods, the product replenishment decision parameters are dynamically adjusted in combination with user group characteristic information, including the user's gender, age and occupation, to optimize the final product replenishment strategy. Specifically, the weights of sales volume, repurchase rate, gross profit margin, dynamic sales rate and new product rate are modified for different user groups; for example, for young users, the weights of new product rate and dynamic sales rate are appropriately increased to meet their preference for new products and hot-selling goods in the market; for older users, the weight of repurchase rate is increased to match their demand for a stable supply of goods; in addition, it is necessary to combine the user's occupational characteristics, such as white-collar users pay more attention to high-gross-profit goods, while freelancers prefer high-dynamic sales rate goods, to further optimize the replenishment decision parameters; by calculating the final product target demand based on the initial demand, to ensure that the replenishment plan recommended to the replenishment staff is more in line with actual needs.

[0063] In one embodiment, the modifying the initial demand for the commodity according to the user group characteristic information and a plurality of preset commodity replenishment decision parameters to determine the target demand for the commodity includes: According to the user group characteristic information, weight adjustment is performed on the commodity replenishment decision parameters to obtain the target weights corresponding to the adjusted commodity replenishment parameters; Specifically, according to the user group characteristics, including gender, age and occupation, the weights of commodity replenishment decision parameters (sales volume, repurchase rate, gross profit margin, turnover rate and new product rate) are dynamically adjusted to optimize the replenishment decision; different user groups have different concerns about commodities. For example, male users pay more attention to high-sales products and are more likely to accept commodities with high profits but stable quality; female users are more inclined to try new products and have a higher demand for repurchase of commodities with stable supply; young users prefer new products and commodities with high turnover rate, while older users prefer commodities with high repurchase rate; white-collar users may pay more attention to high-gross-profit commodities, while freelancers may pay more attention to hot-selling commodities in the market. Therefore, by analyzing the characteristics of user groups, the weights of various replenishment decision parameters are adjusted to make the replenishment strategy more in line with the needs of target users, thereby improving the accuracy of inventory management and sales efficiency.

[0064] In one embodiment, the weight adjustment of the commodity replenishment decision parameters is performed according to the user group characteristic information, and the target weights corresponding to the adjusted commodity replenishment parameters are obtained, including: According to the user's gender, the weight of one or more parameters in the commodity replenishment decision parameters is adjusted to obtain the gender correction weights corresponding to the commodity replenishment decision parameters; Specifically, for user groups of different genders, the weights of commodity replenishment decision parameters are initially adjusted to reflect the impact of gender differences on commodity demand. For example, female users are usually more sensitive to new product launches and repurchase rates, so when calculating weights, the weights of new product launch rates and repurchase rates should be appropriately increased, while the attention to gross profit margin should be reduced. On the contrary, male users prefer high-gross-profit commodities and pay more attention to sales, so the weights of gross profit margin and sales should be increased. In this process, historical sales data and user purchase preferences are combined to calculate the degree of influence of each commodity replenishment decision parameter in user groups of different genders, and the gender-corrected weights of each commodity replenishment decision parameter are obtained.

[0065] According to the user's age, the gender correction weights are adjusted to obtain the age correction weights; Specifically, after obtaining the gender correction weight, the gender correction weight is further optimized and adjusted in combination with the characteristics of the user's age group to obtain the age correction weight; for example, young users (such as 18-30 years old) are more easily attracted by market trends and new products, so the weights of the new product rate and the dynamic sales rate should be increased, while the impact of the repurchase rate should be reduced. Middle-aged users (such as 30-50 years old) have relatively stable purchasing habits and tend to prefer products with high brand loyalty, so the weight of the repurchase rate should be increased, while the proportion of the new product rate should be reduced; elderly users (over 50 years old) pay more attention to the cost-effectiveness and user experience of the product, so the weights of the gross profit margin and the repurchase rate should be increased, while the impact of the dynamic sales rate should be reduced. By analyzing the consumption habits of different age groups, the gender correction weight is adjusted to obtain the age correction weight, so that the replenishment strategy recommended to the replenishment staff is more in line with the actual needs of users of different age groups.

[0066] According to the user's occupation, each of the age correction weights is adjusted to obtain each of the target weights.

[0067] Specifically, the occupational attributes of users affect their spending power and purchasing preferences. Therefore, on the basis of the age-corrected weights, the weights of replenishment decision parameters should be further adjusted according to occupational characteristics. For example, office white-collar workers tend to buy high-gross-profit products due to their strong spending power, so the gross profit margin weight should be appropriately increased, while the reliance on sales rate should be reduced. Students tend to buy reasonably priced products with high repurchase rates due to their limited budgets, so the weights of repurchase rate and sales volume should be increased, while the weights of gross profit margin and new product rate should be reduced. Through the comprehensive adjustment of occupational factors, the final target weights of the replenishment decision parameters for each product are formed, so that the replenishment strategy recommended to replenishment personnel can more accurately match the needs of different occupational groups.

[0068] According to the weight of each target, a weighted analysis is performed on each commodity replenishment decision parameter to obtain a weighted analysis result; Specifically, after completing the multi-dimensional weight adjustment of gender, age and occupation, the target weight is used to perform weighted calculation on the commodity replenishment decision parameters (sales volume, repurchase rate, gross profit margin, sales rate and new product launch rate) to comprehensively measure the market demand potential of each commodity. The weighted analysis process can consider the weight contribution of different commodities in different user groups. For example, the weight of new products that have a greater impact on the young female user group should be higher, while the gross profit margin that has a greater impact on the middle-aged male user group should account for a higher proportion. By calculating the demand contribution of each commodity in different user groups, a comprehensive weighted analysis result is obtained to provide data support for the replenishment decision ultimately provided to the replenishment personnel.

[0069] The target demand for the product is determined based on the weighted analysis result and combined with the initial demand for the product.

[0070] Specifically, after completing the weighted calculation of the commodity replenishment decision parameters, combined with the initial demand, the target demand for each commodity is finally determined; the initial demand is usually derived from historical sales data and market trend forecasts, while the weighted analysis results are corrected according to the characteristics of the current user group, so the target demand can more accurately reflect the actual market demand. For example, if a certain commodity has a low demand in the initial demand forecast, but after weighted analysis, it is found that the target user group has a high repurchase rate and new product rate, the target demand should be increased accordingly to prevent insufficient inventory. On the contrary, if the initial demand forecast for a certain commodity is high, but after weighted analysis, it is found that the target user group has a low purchasing tendency, the target demand should be reduced to avoid inventory backlogs. Finally, by combining the weighted analysis results and the initial demand, it can help replenishment personnel achieve accurate replenishment.

[0071] According to the business needs of unmanned vending machines, the optimization goals are determined to be product profits and user preference; Specifically, in the actual replenishment process, the goal of the unmanned vending machine is not only to meet the demand for goods, but also to ensure that the replenishment plan is the most optimized for the business, which means that the replenishment decision must not only consider how to meet customer needs (by increasing customer favorability), but also how to bring higher profits (by selecting high-profit goods). Based on business needs, it is necessary to balance these two goals: on the one hand, selecting higher-profit goods will directly improve the profitability of the unmanned vending machine; on the other hand, increasing user favorability can promote customer repeat purchases and improve customer satisfaction, thereby increasing sales and reputation. By setting these two optimization goals, economic benefits and user experience are comprehensively considered in the replenishment process, and ultimately the maximum business value is achieved.

[0072] Obtaining preset constraints, wherein the constraints include inventory capacity constraints corresponding to the current unmanned vending machine, inventory capacity constraints corresponding to the product, replenishment budget constraints, and sales cycle constraints; Specifically, in actual operation, replenishment decisions cannot be separated from the actual operating environment and need to consider multiple constraints. Inventory capacity constraint means that the space of the unmanned vending machine itself is limited and cannot accommodate an unlimited number of goods. Therefore, replenishment must be arranged according to the actual space. The commodity inventory capacity constraint means that there is a quantity limit for each commodity in the vending machine. Some commodities may occupy a large space due to size, packaging, etc., thus affecting the placement of other commodities. The replenishment budget constraint means that the replenishment process will be limited by funds and it is impossible to replenish a large number of each commodity in a short time. The sales cycle constraint means that the goods need to be sold out within a certain sales cycle, so the sales cycle of each commodity must be predicted in order to complete the replenishment within a reasonable time. Taking these constraints into consideration can help replenishment personnel make the most optimized replenishment decision.

[0073] According to the constraint conditions and the optimization objectives, combined with the target demand for the commodity, a multi-objective optimization process is performed on the commodity replenishment quantity; Specifically, a multi-objective optimization algorithm is used to solve the decision-making problem of how to maximize the preset optimization goals (product profit and user popularity) under limited resources (such as inventory capacity constraints, replenishment budget constraints and sales cycle constraints). Multi-objective optimization processing not only needs to consider the target demand for products, but also needs to take into account constraints such as inventory capacity, budget constraints and sales cycle. These multiple goals and constraints are combined through mathematical models to finally calculate the most appropriate product replenishment quantity. For example, in the mathematical logic of multi-objective optimization, a function needs to be defined for each goal first. Usually these goals are mutually competitive, such as maximizing product profit and maximizing user popularity. Assuming there are n products, the optimization goal is to determine the replenishment quantity x1, x2, ..., xn based on the characteristic information of each product. Each objective function is a function fi (x1, x2, ..., xn) about the replenishment quantity. For example, increasing the replenishment quantity of a certain product will lead to a decrease in profit. In order to comprehensively consider these goals, they are converted into a comprehensive objective function through the weighted summation method or Pareto optimization method. Then, combined with constraints such as inventory capacity C, budget B, sales cycle T, etc., a mathematical constraint optimization model is formed: Among them, Maximize means maximization, wi is the weight coefficient of each goal, xi is the replenishment quantity of the ith product, indicating the importance of each goal, and the constraints include the maximum inventory capacity constraint of the product xi≤Ci, Ci is the inventory capacity constraint of the ith product, indicating the maximum inventory of the product. Budget constraint∑ i=1 n p i xi≤B, B is the total budget constraint, which indicates the maximum amount of funds that can be used during the replenishment process; and the replenishment quantity should not exceed the maximum value of the sales cycle xi≤Ti, Ti is the sales cycle constraint of the ith product, which refers to the sales cycle of the product, that is, how long it takes to replenish. By solving this optimization problem, a set of optimal product replenishment quantities x1, x2, ..., xn are obtained, which not only meet the constraints but also maximize the comprehensive objective function. The specific optimization method uses linear programming, integer programming or genetic algorithm, etc. The specific choice depends on the form of the objective function and constraints. Finally, through multi-objective optimization, the system can balance different objectives while meeting business needs and find the most suitable product replenishment plan.

[0074] According to the optimization results, select the target products from the preset product list and replenish them.

[0075] Specifically, after the multi-objective optimization calculation, the products that need to be replenished are screened from the preset product list according to the calculation results, ensuring that each product is put on the shelf according to the optimized quantity. This not only improves the efficiency of the replenishment process, but also ensures that the products in the sales cabinet meet market demand and business goals. By tracking and adjusting the replenishment list in real time, the product replenishment can be flexibly carried out according to the changing sales data and market trends, further optimizing the operating efficiency of the sales cabinet.

[0076] In one embodiment, see Figure 8 , the step of selecting target products for replenishment from a preset product list according to the optimization processing result includes: According to the optimization processing result, determining whether the current inventory capacity of the target product meets the replenishment demand; Specifically, by analyzing the optimization processing results, we first evaluate whether the current inventory capacity of the target product is sufficient to meet the replenishment quantity provided by the demand forecasting model. If the inventory capacity of the target product is less than the replenishment demand, we identify this deficiency and decide whether to replenish it. The satisfaction degree of inventory capacity is crucial to optimizing the replenishment plan. Only when the inventory capacity is not met will it trigger further commodity substitution and adjustment processes to ensure the smooth replenishment.

[0077] When it is determined that the inventory capacity does not meet the replenishment demand, feature extraction is performed on each non-target product in the preset product list to determine feature information of the non-target product; Specifically, when it is determined that the inventory capacity of the target product cannot meet the replenishment demand, feature extraction will be performed on all non-target products in the preset product list. This process includes extracting features from the product's attribute information (such as category, price, sales status, user ratings, etc.) and converting these features into a processable form in the data model. The feature information of non-target products will help in subsequent similarity calculations, allowing evaluation of whether these non-target products can replace the target products to meet the replenishment demand without significantly affecting the ultimate business goals.

[0078] Calculating similarity between the commodity feature information and the non-target commodity feature information, determining the similarity between each non-target commodity and the target commodity, and obtaining the inventory capacity corresponding to each non-target commodity; Specifically, by calculating the similarity between the target product and each non-target product, it is determined which non-target products are more similar to the target product in terms of characteristics. The similarity calculation is usually based on the multi-dimensional characteristics of the product, such as color, brand, category, price, etc., using methods such as cosine similarity and Euclidean distance to quantify the degree of similarity between the two. This calculation helps determine which products can be used as alternative options when the target product is out of stock, thereby reducing business losses caused by insufficient inventory.

[0079] Compare the similarities between each non-target product and the target product to determine the maximum similarity; Specifically, based on the similarity calculated previously, the actual inventory capacity information is extracted for each non-target product. Inventory capacity is crucial for replenishment decisions, as it allows knowing the available quantity of each product in actual sales. In this step, the existing inventory situation of the product is used to evaluate whether more products can be added or the target product can be replaced. By combining it with inventory capacity, the replenishment strategy is further refined to ensure that there is no over-replenishment or excess inventory.

[0080] Compare the maximum similarity with a preset similarity threshold to determine a comparison result; Specifically, the calculated maximum similarity value is compared with a pre-set similarity threshold, which is a predefined standard that indicates whether the similarity between products is high enough to ensure that the replacement product matches the target product to a certain degree in terms of characteristics. By comparison, it is determined which non-target products are similar enough to be substitutes for the target product. If the maximum similarity is greater than or equal to the similarity threshold, it means that the non-target product is similar to the target product in terms of characteristics and can be used as a candidate for replenishment. Otherwise, the non-target product will be abandoned as a replacement option.

[0081] According to the comparison result, when the maximum similarity is greater than the similarity threshold, comparing the non-target commodity inventory capacity corresponding to the maximum similarity with a preset capacity threshold; Specifically, once the non-target products corresponding to the maximum similarity are determined, the inventory capacity of these products is checked. Specifically, the inventory capacity of the non-target products is compared with the preset capacity threshold. The capacity threshold refers to the minimum inventory requirement of each product, indicating whether the product has sufficient inventory for replenishment. If the inventory capacity of the non-target product is greater than the preset capacity threshold, it means that the product has the potential to meet the replenishment demand and can be used as a replenishment object; otherwise, the replenishment requirements are not met and other replenishment options will be selected.

[0082] When the inventory capacity of the non-target commodity is greater than the capacity threshold, the target commodity and the non-target commodity corresponding to the maximum similarity are screened out from the preset commodity list for replenishment.

[0083] Specifically, based on the comparison results of the first two steps, it will be decided which products need to be replenished. If the maximum similarity is greater than the similarity threshold and the inventory capacity of the non-target products exceeds the capacity threshold, the target products and non-target products corresponding to the maximum similarity will be screened out from the preset product list for replenishment. These products will be put on the shelves together to ensure that user needs can be met when inventory is insufficient to avoid out-of-stock situations. At the same time, through the introduction of products with the maximum similarity, users' acceptance of and possibility of purchasing alternative products will be improved, thereby maximizing commercial benefits.

[0084] Example 2

[0085] In actual operation, unmanned vending machines are widely distributed in different areas such as office buildings, schools, shopping malls, subway stations, etc. The target demand for goods in each vending machine varies due to different geographical locations, customer flow and consumption habits; the traditional replenishment method is often based on a fixed time or inventory warning, which causes some vending machines to be sold out in advance, affecting sales, or replenishing goods too frequently, increasing operating costs; therefore, in order to intelligently calculate a global optimal replenishment time point, on the basis of Example 1, after inputting the commodity feature information into the commodity demand prediction target model and outputting the commodity target demand, it also includes: Obtain target commodity target demand and location information of each target area corresponding to the unmanned vending machine in each target area; Specifically, the method of Example 1 is used to obtain the target demand for the target goods corresponding to the unmanned vending machines in each target area; GIS (Geographic Information System) or GPS positioning technology is used to obtain the geographical location of each vending machine and divide it into corresponding areas for subsequent regional replenishment optimization. By dividing the areas, a replenishment plan can be formulated more specifically to improve the efficiency of logistics distribution.

[0086] According to the target demand of each target commodity, combined with historical sales data and real-time inventory information, the corresponding replenishment demand of each unmanned vending machine is calculated; Specifically, combined with historical sales data, time series analysis (such as ARIMA, LSTM neural network) is used to predict the target demand for goods in the future; the current inventory level is calculated, and if the inventory is lower than a certain set threshold, such as the safety stock, the replenishment demand calculation is triggered; a replenishment model, such as the inventory control model EOQ, is used to calculate the replenishment required for each vending cabinet to ensure sufficient supply of goods while avoiding excessive inventory; combined with external factors such as holidays, weather, and time periods, the demand is dynamically adjusted. For example, when the temperature rises, the demand for beverages may increase, and the predicted replenishment quantity needs to be increased accordingly; through historical data analysis and real-time inventory calculation, the replenishment demand is accurately calculated to ensure that the replenishment quantity can meet sales demand without causing backlogs; combined with external environmental factors to optimize predictions, improve system adaptability, and enhance the service capabilities of unmanned vending cabinets; machine learning is used to optimize inventory management, reduce corporate operating costs, and increase profit margins.

[0087] According to the location information of each target area, obtaining the sales cycle corresponding to each target area; Specifically, the sales data of each unmanned vending machine in different time periods, such as hours, days, and weeks, are counted to analyze sales trends; the K-means clustering algorithm is used to classify vending machines with similar sales patterns into the same category to determine the sales cycle of each area; for example, unmanned vending machines in office areas have strong sales during the day on weekdays, while unmanned vending machines in residential areas have higher demand at night; combined with the classification of business districts (such as commercial districts, schools, stations, etc.), the sales cycle is further refined to adapt to the sales patterns of different environments. By analyzing the sales cycle, the rationality of the replenishment time is improved, and the replenishment plan is more in line with the actual sales situation; the clustering algorithm is used to improve the accuracy of the regional sales cycle and optimize the operation and management of the unmanned vending machine; combined with the classification of business districts and historical data, the prediction ability is improved to make the replenishment plan more flexible.

[0088] Calculate the replenishment frequency according to the replenishment demand and the sales cycle, and determine the optimal replenishment cycle corresponding to each unmanned vending machine; Specifically, according to the replenishment demand and the sales cycle, an optimization algorithm, such as integer linear programming and dynamic programming, is used to calculate the optimal replenishment cycle of each unmanned vending machine; multiple constraints are set, such as inventory threshold, product shelf life, transportation cost, etc., to ensure the rationality of the replenishment cycle; the replenishment frequency of different products is calculated according to the consumption rate of different products; for example, the replenishment cycle of high-selling products should be shorter, while the replenishment cycle of low-selling products can be appropriately extended; the replenishment cycle is adjusted dynamically to adapt to the actual operation situation. By calculating the optimal replenishment cycle, unnecessary replenishment is reduced, logistics costs are reduced, and supply chain efficiency is improved; multiple factors are combined for optimization to improve the inventory turnover rate of the vending machine and avoid product backlog or out-of-stock. A dynamic adjustment mechanism is adopted to improve the flexibility of the system and make the replenishment plan more adaptable.

[0089] According to the optimal replenishment cycle of each vending machine, the replenishment time of the vending machines in the same area is optimized synchronously to determine the optimal replenishment time window at the regional level; Specifically, a distributed computing method is used to comprehensively calculate the replenishment cycles of all vending machines in the same area to determine the optimal replenishment time window; a genetic algorithm or a heuristic optimization algorithm is used to find the optimal solution among multiple replenishment plans to ensure that the replenishment time is reasonably arranged; the replenishment time window is further optimized in combination with the operating time of the replenishment vehicle and the work schedule of the replenishment staff to reduce the replenishment cost and time waste; real-time data monitoring (such as inventory changes, traffic conditions, etc.) is used to dynamically adjust the replenishment time window to respond to emergencies (such as temporary out-of-stock, peak demand surges, etc.); regional-level replenishment optimization is used to reduce the number of replenishment times and improve logistics distribution efficiency; an intelligent scheduling algorithm is used to improve the rationality of the replenishment plan and avoid waste of resources; combined with real-time monitoring, the system adaptability is improved to reduce operational risks caused by emergencies.

[0090] According to the optimal replenishment time window at the regional level, the replenishment paths between different regions are optimized to determine the optimal replenishment strategy, wherein the replenishment strategy includes replenishment time points and replenishment routes.

[0091] Specifically, a path optimization algorithm is used to calculate the shortest replenishment path to improve transportation efficiency. In combination with real-time traffic data, the replenishment route is dynamically adjusted to avoid traffic jams. The operation plan of the replenishment vehicle is optimized to reduce transportation costs. For example, priority is given to replenishing vending machines in high-sales areas or those with severe out-of-stock situations to ensure that replenishment resources are reasonably allocated. Through path optimization, replenishment efficiency is improved and vehicle operating costs are reduced. Real-time data adjustments are used to increase the flexibility of replenishment plans and reduce replenishment delays. In combination with an intelligent scheduling system, the overall operational efficiency of the vending machines is improved, thereby increasing corporate profitability.

[0092] Example 3 See also Fig. 9Embodiment 3 of the present invention further provides a retail terminal inventory replenishment device based on multimodal data, the device comprising: An image acquisition module, used to decompose the real-time video data of the unmanned vending machine in the commodity transaction scenario into multiple frames of real-time images; The target detection module is used to perform commodity target detection on each frame of the real-time image, and determine the commodity area image according to the target detection result; A multimodal feature extraction module, used to extract multimodal features from the commodity area image to determine commodity feature information; A human feature extraction module is used to extract human features from user area images in historical commodity transaction scenarios and update user group feature information; A target model training module, used for inputting the user group characteristic information as incremental data into a pre-trained commodity demand forecasting initial model for fine-tuning training, and obtaining a fine-tuned commodity demand forecasting target model; The replenishment and shelf-putting module is used to input the commodity feature information into the commodity demand prediction target model, and select the target commodity for replenishment and shelf-putting from the preset commodity list according to the prediction result.

[0093] Specifically, the retail terminal inventory replenishment device based on multimodal data provided by the embodiment of the present invention can reflect changes in customer behavior in real time, avoid delays in commodity replenishment or over-replenishment, and thus achieve accurate and intelligent replenishment management.

[0094] Example 4

[0095] Embodiment 4 of the present invention further provides a retail terminal inventory replenishment system based on multimodal data, the system comprising: The first camera 1 and the second camera 2; the first camera 1 has a collection range inside and outside the smart container, and the second camera 2 has a collection range outside the unmanned vending machine; Fig.10 A schematic diagram of the structure of an intelligent retail terminal provided in Embodiment 4 of the present invention is shown.

[0096] A retail terminal inventory replenishment system based on multimodal data may include a processor and a memory storing computer program instructions.

[0097] Specifically, the processor may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.

[0098] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside the data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0099] The processor implements any one of the retail terminal inventory replenishment methods based on multimodal data in the above embodiments by reading and executing computer program instructions stored in the memory.

[0100] In one example, the retail terminal inventory replenishment system based on multimodal data may also include a communication interface and a bus. Fig.11 As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.

[0101] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.

[0102] The bus includes hardware, software or both, and the components of the device are coupled to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnection, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnection, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, the bus may include one or more buses. Although the embodiments of the present invention describe and illustrate a specific bus, the present invention contemplates any suitable bus or interconnection.

[0103] In summary, the embodiments of the present invention provide a retail terminal inventory replenishment method, device and system based on multimodal data.

[0104] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0105] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0106] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0107] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A retail terminal inventory replenishment method based on multimodal data, characterized in that: The method comprises: According to the real-time video data of the unmanned vending machine in the commodity transaction scenario, the real-time video data is decomposed into multiple frames of real-time images; Perform commodity target detection on each frame of the real-time image, and determine a commodity area image according to the target detection result; Performing multimodal feature extraction on the product area image to determine product feature information; Extract human features from user area images in historical commodity transaction scenarios and update user group feature information; Inputting the user group characteristic information as incremental data into a pre-trained commodity demand forecasting initial model for fine-tuning training to obtain a fine-tuned commodity demand forecasting target model; The commodity feature information is input into the commodity demand forecasting target model, and target commodities are selected from a preset commodity list for replenishment and listing according to the forecasting result.

2. The retail terminal inventory replenishment method based on multimodal data according to claim 1, characterized in that: The performing commodity target detection on each frame of the real-time image and determining the commodity area image according to the target detection result comprises: Resize and denoise each frame of the real-time image to determine multiple frames of target images after denoising; Inputting the target image of each frame into a pre-trained target detection model to determine initial target boundary position information, wherein the initial target includes a product target and a user target; Acquiring an initial target area image within the initial target boundary according to the initial target boundary position information; The commodity label detection is performed on the initial target area image, and the commodity area image is determined according to the detected commodity label position information and the initial target boundary position information.

3. The retail terminal inventory replenishment method based on multimodal data according to claim 1, characterized in that: The performing multimodal feature extraction on the commodity area image to determine commodity feature information includes: Performing color space conversion on the commodity area image, and obtaining a color histogram of the commodity area in the converted color space; Aggregating and optimizing the color histogram to determine color feature information corresponding to the product area; Performing edge detection on the commodity area image, and determining contour feature information of the commodity area according to the edge detection result; Extracting texture features from the commodity area image to determine texture feature information of the commodity area; Performing text recognition on the commodity area image to determine text information in the commodity area; Performing natural language processing on the text information to determine text feature information; Multimodal data fusion is performed on the color feature information, the contour feature information, the texture feature information and the text feature information to determine the commodity feature information.

4. The retail terminal inventory replenishment method based on multimodal data according to claim 3, characterized in that: The performing multimodal data fusion on the color feature information, the contour feature information, the texture feature information and the text feature information to determine the commodity feature information includes: According to the contour feature information, the product area image is segmented to determine the local area position information corresponding to the product body; Extracting local area color feature information from the color feature information according to the local area position information; Extracting local area contour feature information from the contour feature information according to the local area position information; Extracting local area texture feature information from the texture feature information according to the local area position information; Extracting local area text feature information from the text feature information according to the local area position information; Inputting the commodity area image into a pre-trained target classification model to determine the commodity category; Determine, according to the commodity category and the mapping relationship table between the preset commodity category and the weight, a first weight of the local area color feature information, a second weight of the local area contour feature information, a third weight of the local area texture feature information, and a fourth weight of the local area text feature information; According to the first weight, the second weight, the third weight and the fourth weight, the local area color feature information, the local area contour feature information, the local area texture feature information and the local area text feature information are weightedly fused, and the fused feature information is determined as the product feature information.

5. The retail terminal inventory replenishment method based on multimodal data according to any one of claims 1 to 4, characterized in that: The extracting of human features from user area images in historical commodity transaction scenarios and updating user group feature information includes: Obtain historical images of commodity trading scenarios; Performing human body key point detection on the historical image to determine the user area image; Performing image enhancement processing on the user area image to determine an enhanced user target image, wherein the image enhancement processing includes super-resolution reconstruction, image denoising and contrast adjustment; Inputting the user target image into a pre-trained posture detection model to determine posture feature information; Performing clothing type recognition on the user target image to determine clothing feature information of the user; Preliminarily predicting the age stage and gender of the user based on the clothing feature information and the user posture feature information, and determining an initial age range and user gender; Performing face detection on the user target image to determine whether there is a user face in the user target image; If the user's face exists, facial feature extraction is performed on the facial region image to determine facial feature information, wherein the facial feature information includes eye feature information and facial texture feature information; According to the eye feature information and the facial texture feature information, the initial age range is corrected for a second time, and the corrected age range is determined as the target age range; According to the geographical location information of the unmanned vending machine and the clothing feature information, the occupation of the user is predicted and determined; The user group characteristic information is determined according to the user gender, target age range and the user occupation.

6. The retail terminal inventory replenishment method based on multimodal data according to claim 5, characterized in that: The inputting the user group characteristic information as incremental data into the pre-trained commodity demand forecasting initial model for fine-tuning training to obtain the fine-tuned commodity demand forecasting target model includes: Using the pre-collected historical commodity sales data and historical commodity feature information corresponding to each commodity, the long-short term memory model is trained to determine the initial commodity demand forecasting model; Classifying the target age range to determine age characteristic information corresponding to the target age range; Classifying and encoding the user's occupation to determine occupational characteristic information; Converting and encoding the user's gender to determine binary feature information after processing; According to preset weights, weighted fusion is performed on the age feature information, the occupation feature information and the binary feature information to determine multi-dimensional feature information; The multi-dimensional feature information is input into the commodity demand forecasting initial model for incremental training and model fine-tuning to determine the commodity demand forecasting target model.

7. The retail terminal inventory replenishment method based on multimodal data according to claim 6, characterized in that: The step of inputting the commodity feature information into the commodity demand forecasting target model and selecting target commodities for replenishment and shelf placement from a preset commodity list according to the forecasting result comprises: Inputting the commodity feature information into the commodity demand forecasting target model, and outputting the commodity target demand; Obtaining an optimization target in a commodity transaction scenario, wherein the optimization target includes commodity profit and user preference; Obtaining preset constraints, wherein the constraints include inventory capacity constraints corresponding to the current unmanned vending machine, inventory capacity constraints corresponding to the product, replenishment budget constraints, and sales cycle constraints; According to the constraint conditions and the optimization objectives, combined with the target demand for the commodity, a multi-objective optimization process is performed on the commodity replenishment quantity; According to the optimization results, select the target products from the preset product list and replenish them.

8. The retail terminal inventory replenishment method based on multimodal data according to claim 7, characterized in that: The step of selecting target products for replenishment from the preset product list according to the optimization processing result includes: According to the optimization processing result, determining whether the current inventory capacity of the target product meets the replenishment demand; When it is determined that the inventory capacity does not meet the replenishment demand, feature extraction is performed on each non-target product in the preset product list to determine feature information of the non-target product; Calculating similarity between the commodity feature information and the non-target commodity feature information, determining the similarity between each non-target commodity and the target commodity, and obtaining the inventory capacity corresponding to each non-target commodity; Compare the similarities between each non-target product and the target product to determine the maximum similarity; Compare the maximum similarity with a preset similarity threshold to determine a comparison result; According to the comparison result, when the maximum similarity is greater than the similarity threshold, comparing the non-target commodity inventory capacity corresponding to the maximum similarity with a preset capacity threshold; When the inventory capacity of the non-target commodity is greater than the capacity threshold, the target commodity and the non-target commodity corresponding to the maximum similarity are screened out from the preset commodity list for replenishment.

9. A retail terminal inventory replenishment device based on multimodal data, characterized in that: The device comprises: An image acquisition module, used to decompose the real-time video data of the unmanned vending machine in the commodity transaction scenario into multiple frames of real-time images; The target detection module is used to perform commodity target detection on each frame of the real-time image, and determine the commodity area image according to the target detection result; A multimodal feature extraction module, used to extract multimodal features from the commodity area image to determine commodity feature information; A human feature extraction module is used to extract human features from user area images in historical commodity transaction scenarios and update user group feature information; A target model training module, used for inputting the user group characteristic information as incremental data into a pre-trained commodity demand forecasting initial model for fine-tuning training, and obtaining a fine-tuned commodity demand forecasting target model; The replenishment and shelf-putting module is used to input the commodity feature information into the commodity demand prediction target model, and select the target commodity for replenishment and shelf-putting from the preset commodity list according to the prediction result.

10. A retail terminal inventory replenishment system based on multimodal data, characterized in that: include: A first camera and a second camera; the collection range of the first camera is inside and outside the unmanned vending cabinet, and the collection range of the second camera is outside the unmanned vending cabinet; At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 8.

Images