Commodity recommendation method based on image recognition, electronic equipment, storage medium and application

Abstract

The invention provides a commodity recommendation method based on image recognition, an electronic device, a storage medium and an application, and belongs to the technical field of commodity recommendation. An offline commodity sales place obtains attention video information of each customer on a commodity through a video, and divides the video into a plurality of video segments according to offline commodity sales partitions; obtaining a plurality of marked graphs from each video segment, summarizing the marked graphs of each customer to obtain a customer intention commodity graph, constructing a third-order learning neural network model, inputting the marked graphs into the third-order learning neural network model, and carrying out recognition to obtain a commodity concerned by each customer and detailed information of the concerned commodity, and recommending the commodities to the corresponding clients from the platforms related to the online and offline commodity sales places. The customer behavior information of the offline sales place is collected, then image recognition is carried out, the refined concern point of each action of the customer is recognized, and then required commodities can be accurately recommended to the customer.

Description

Technical Field

[0001] The present invention relates to the technical field of product recommendation, and particularly to a product recommendation method, an electronic device, a storage medium and an application based on image recognition. Background Art

[0002] Big data, a data set characterized by large capacity, multiple types, fast access speed and high application value, was first applied in the IT industry and is currently rapidly developing into a new generation of information technology and service mode for collecting, storing and associatively analyzing data with huge quantity, scattered sources and diverse formats to discover new knowledge, create new value and enhance new capabilities. Big data must adopt a distributed architecture for distributed data mining of massive data, so it must rely on distributed processing of cloud computing, distributed databases, cloud storage and virtualization technologies.

[0003] Currently, in order to attract users, many merchants will set up display screens at the store entrance. The display screens will continuously cycle through the display of products to attract the attention of users. However, the product content displayed in this way is very limited. Usually, only promotional products, special offer products, etc. will be displayed. In fact, people come and go at the store entrance, and different users often have various preferences for products. Simply cycling through the display of promotional products and special offer products can only attract a small number of users. There are still a large number of users who are not attracted because the products they prefer are not displayed, resulting in the current offline product recommendation display system not being very effective.

[0004] With the development of big data technology, it has been gradually applied to consumer product recommendation. For example, Chinese Patent No. CN111028029B discloses an offline product recommendation method, device and electronic device. Chinese Patent No. CN113435936B discloses an online and offline consumer product recommendation method and system based on big data. Chinese Patent Application No. CN108564414A discloses a product recommendation method and system based on offline behavior. These recommendation systems only rely on the basic information of customers and some interaction or browsing information, and cannot identify which products the customers are specifically interested in. At the same time, they cannot identify the reasons why the customers have not purchased the products they are interested in, and cannot make targeted recommendations based on the customer's concerns when making recommendations. At the same time, when it is necessary to recognize all the images of the customers' behaviors, a large amount of image data needs to be processed, resulting in low recognition efficiency. Therefore, it is necessary to design a product recommendation method, an electronic device, a storage medium and an application based on image recognition. Summary of the Invention

[0005] The object of the present invention is to provide a product recommendation method, an electronic device, a storage medium and an application based on image recognition, so as to solve the technical problems that the existing product recommendation methods cannot recommend according to the customer's focus and have low efficiency in processing data.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A product recommendation method based on image recognition, the method includes the following steps:

[0008] Step 1: The offline product sales venue obtains the attention video information of each customer for the product through video.

[0009] Step 2: Divide the video into several video segments according to the offline product sales area.

[0010] Step 3: Obtain several marker maps from each video segment.

[0011] Step 4: Summarize the marker maps of each customer to obtain a customer-intended product map.

[0012] Step 5: Construct a third-order learning neural network model.

[0013] Step 6: Input the marker map into the third-order learning neural network model to identify the products and the detailed information of the concerned products that each customer is interested in.

[0014] Step 7: Recommend the products to the corresponding customers from the platforms related to the online and offline product sales venues.

[0015] Further, the specific process of Step 1 is: Install a customer behavior video camera at the offline product sales venue to obtain the behavior and voice of the customer when purchasing products at the product sales venue. The video camera performs real-time positioning and tracking according to the number and location of the customers. The obtained video includes the process of the customer purchasing products and discussing products. Then, the video and voice are obtained simultaneously to obtain the entire video of the customer's product purchasing process at the offline product sales venue.

[0016] Further, the specific process of Step 2 is: Taking the customer as the clue, starting from when the customer enters the offline product sales venue and ending when the customer leaves the offline product sales venue, divide the video into different video segments according to each product sales area where the customer is located. Then, the title of the video segment is the customer information and the product area information. Summarize the videos within the same product area into a group to obtain a video segment group according to the product area. Then, summarize the video segment groups of all users to obtain a video segment set according to different product areas.

[0017] Further, the specific process of step 3 is as follows: In video marking, taking a single piece of merchandise as the smallest unit, from the moment a customer notices a certain piece of merchandise until the customer completely leaves the corresponding merchandise is considered a screenshot segment. Different video screenshots are intercepted according to actions, and then the information of the corresponding voice is read and recognized as text. Then, the screenshot image and the voice text information are used as the marked image. The image of the screenshot is marked with action cues to obtain the customer's merchandise attention image dataset. By later identifying the action images, attention images, and voice information of the customer's attention merchandise, the specific attention points of the customer's attention to a certain piece of merchandise are obtained. For the merchandise that has been purchased, the reasons and demand points for purchase are obtained. If not purchased yet, it is determined whether the detailed features of the customer's attention merchandise do not meet the requirements, the price does not meet the requirements, or the customer's own needs have not been met. If a customer does not notice a certain piece of merchandise when browsing a physical merchandise sales venue, the corresponding video is discarded.

[0018] Further, the specific process of step 5 is as follows:

[0019] Step 5.1: Set up a perceptron for classifying image nodes. Use a two-layer MLP for image node classification. The activation function of the first layer is ReLU, and a normalization layer and Dropout are used to prevent overfitting. The activation function of the second layer is Softmax, which converts the image node representation into the probability of class prediction. The expression is as follows:

[0020] Z = Dropout(LN(σ(XW 0 ))

[0021]

[0022] where X is the original image node representation, σ() represents the activation function. In the first layer of the neural network, ReLU is used. LN() represents layer normalization operation, and Dropout() represents randomly discarding neurons. The two operations work together to prevent overfitting and enable the model to have better generalization ability. And W 0 , W 1 represent the trainable parameters of the two-layer neural network, which map the input image node representation to the prediction probabilities of different classes, that is, Y, to obtain the prediction result;

[0023] Step 5.2: Design the first-level contrast learning module to introduce more edge information to enhance the effect of contrast learning. The first-level contrast learning module is an edge-level contrast learning module. More edge information is introduced through the construction of a multi-hop graph. The multi-hop graph is constructed based on the following rules:

[0024]

[0025] In the construction of a multi-hop graph, if image node a is a neighbor of image node b, and at the same time image node c is a neighbor of image node b, it is considered that image node a is a neighbor of image node c, and an edge is established. More edge information is introduced into the contrast learning to improve the accuracy of image node prediction;

[0026] The loss function of edge-level contrast learning is defined as follows:

[0027]

[0028] where B represents the batch size, represents the neighbor set of image node i, and sim(z i , z j ) represents the similarity between image nodes i and j. τ is the temperature parameter. In the multi-hop graph , if there is an edge between two image nodes, their representations will be closer, that is, the similarity sim(z i , z j ) increases. Through edge-level contrast learning, similar image nodes will be more clustered, while dissimilar image nodes will move away from each other. The contrast learning method can make the distribution of image nodes more compact, thus making the decision boundary of the classifier clearer and finally effectively improving the prediction accuracy;

[0029] Step 5.3: Set the second-level contrast learning module to generate the perturbed feature representation to further optimize the decision boundary of image node classification. The second-level contrast learning module is the feature-level contrast learning module. By using the randomness of Dropout to perturb the image node representation, the stability of the model is strengthened. The specific formula is as follows:

[0030] Z + = Dropout(LN(σ(XW 0 )))

[0031] where Z + is the perturbed image node representation generated by Dropout. In addition, Z + and Z form a contrast learning sample pair, and the following loss function is used to maintain the stability of model training:

[0032]

[0033] where The goal of the loss function is to enhance the stability and robustness of the model by constraining the similarity between the original representation Z and the perturbed representation Z + . Even when the image node features are perturbed, the model can still maintain the reliability of the output results;

[0034] Step 5.4: Set the third-level contrastive learning module to introduce a way of disturbing the order of image nodes to achieve the learning of global information without adding additional parameters, effectively controlling the complexity of the model. The third-level contrastive learning module is an image node-level contrastive learning module. The specific process is as follows:

[0035] Z - =shuffle(Z)

[0036] Among them, shuffle() means to disrupt the order of image nodes to generate negative samples Z - In contrastive learning, the constrained image node representation Z and the perturbed image node representation Z - similarity, so as to achieve the goal of keeping the two away from each other;

[0037] The loss function for image node-level contrastive learning is defined as:

[0038]

[0039] There are two cases. The first is image nodes of different categories. When the perturbed image nodes come from different categories, they should obviously be kept away from each other to enhance the distinction between categories. The second is image nodes of the same category. Even if the perturbed image nodes come from the same category, they are still required to be kept away from each other. This is because each image node has its own unique characteristics. By keeping away from other image nodes of the same category, the model can not only capture the commonalities between categories, but also mine the identifying information of the image nodes themselves.

[0040] Step 5.5: Design a total loss function to train the neural network. The total loss function is as follows:

[0041]

[0042] Among them, α, β, and γ are three adjustable hyperparameters. By adjusting these three parameters, we can balance the effects of different loss terms in the model training process and complete the model construction.

[0043] Furthermore, in step 6, the customer's attention to a product is taken as the minimum recognition unit, and the label map and voice information related to a product are input into the third-order learning neural network model, and the customer's actions and eye gaze are used as recognition image nodes, and then image recognition is performed, and the image features are classified, and then the customer's attention points on the product are predicted. Through the model output, each customer's action is identified to identify the attention to the product price, size, function, appearance, color, style and own needs;

[0044] When a customer views the price information of a product but does not purchase it, it is determined that the price of the product is a concern of the customer. When the price of the product changes, it is used as the main recommendation point for the corresponding customer. When a customer views the detailed information of a product but does not purchase it, it is determined that the product information cannot fully satisfy the customer. When similar products or updated products appear, the similar products or updated products are recommended to the customer. If the customer purchases the product, the customer information and the information of the purchased product are identified, and it is determined that the corresponding category of customers has a demand to purchase the corresponding or related products, and the products are recommended to the corresponding category of customers. When the product that the user views or pays attention to does not have a price or iterative product, the product that the customer pays attention to or views will be recommended to the customer.

[0045] Furthermore, in step 7, the merchant will set up a corresponding mini-program or video to push the products that need to be recommended to the customer's mobile phone or computer.

[0046] An electronic device comprises a video acquisition device, a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the commodity recommendation method based on image recognition as described in any one of claims 1 to 7 are implemented.

[0047] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps in the commodity recommendation method based on image recognition as described in any one of claims 1 to 7 are implemented.

[0048] An application of a product recommendation method based on image recognition, which is applied to online recommendations for all products sold offline, including department store product recommendations, clothing store product recommendations, electrical appliance supermarket product recommendations, and gold shop product recommendations.

[0049] The present invention has the following beneficial effects due to the adoption of the above technical solution:

[0050] (1) The present invention collects customer behavior information at offline sales locations, and then uses image recognition to identify the detailed focus of each customer action, and then accurately recommends the required products to the customer. At the same time, based on the detailed identification of the customer's focus on price, style or other detailed points, accurate change recommendations are made based on the customer's detailed focus, so that the customer transaction rate is higher, and online recommendations are better combined with offline behavior;

[0051] (2) The three - order learning neural network model set in this application can achieve efficient processing of more image data. Firstly, edge - level contrast learning helps the model effectively learn the adjacent relationships of nodes by capturing local graph information. Secondly, node - level contrast learning focuses on mining global information to improve the model's understanding ability of the overall node distribution. Finally, feature - level contrast learning forces the model to capture the internal patterns of global and local information by perturbing node features, enhancing the model's robustness in a noisy environment. The three contrast learning methods complement each other, jointly optimizing the classification boundary of the MLP and significantly improving the model's expressive ability and classification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is the flowchart of the method of the present invention;

[0053] Figure 2 is the flowchart of constructing the three - order learning neural network model of the present invention;

[0054] Figure 3 is the schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] To make the objectives, technical solutions and advantages of the present invention more clearly understood, the following provides preferred embodiments with reference to the accompanying drawings and further elaborates on the present invention. However, it should be noted that many details listed in the specification are only for enabling the reader to have a thorough understanding of one or more aspects of the present invention, and these aspects of the present invention can be implemented even without these specific details.

[0056] Such as Figure 1 shown, a commodity recommendation method based on image recognition, the method includes the following steps: taking the sales of gold and silver jewelry stores as an example:

[0057] Step 1: Obtain video information of each customer's attention to the product through video at offline product sales venues. Install customer behavior video cameras at offline product sales venues to obtain the customer's behavior and voice when purchasing products at the product sales venues. The video cameras are located and tracked in real time according to the number and location of customers. The obtained video includes the process of customers purchasing and discussing products. Then, the video and voice are obtained at the same time to obtain the video of the customer's entire process of purchasing products at offline product sales venues. By installing cameras at the top, the customer's attention to jewelry after entering the jewelry store is obtained through video, and then the customer's process and voice of selecting items are recorded. For example, when the customer is hesitating or hesitating about the price, it is judged that the customer is considering the price of the item. When the price of the item changes, a recommendation notification is made. If the overall structure of the decoration, such as the overall appearance when wearing it, is not very satisfied, the next iteration product is made, and the focus is on recommendation and notification. Or if the size does not meet the customer's requirements or the customer's intended product category direction is identified from voice communication, when the corresponding product appears, it is recommended in a timely manner.

[0058] Step 2: Divide the video into several video segments according to offline product sales zones. Taking the customer as the clue, the video starts when the customer enters the offline product sales venue, and ends when the customer leaves the offline product sales venue. The video is divided into different video segments according to each product sales zone where the customer is located. The title of the video segment is the customer information and the product partition information. The videos in the same product partition are aggregated into a group to obtain a video segment group according to the product partition. Then, the video segment groups of all users are aggregated to obtain a video segment set according to different product partitions. For example, a pendant is a partition, a ring is a partition, and bracelets, earrings, etc. are all partitions. Then, based on the customer's expression and actions when viewing the product, identify what the customer is concerned about.

[0059] Step 3: Obtain several marked images from each video segment. In video marking, taking a single item as the smallest unit, from the moment a customer focuses on a certain item to the moment the customer completely leaves the corresponding item is considered as a screenshot segment. Different video screenshots are intercepted according to actions, and then the information of the corresponding voice is recognized as text. Then, the screenshot image and the voice text information are used as marked images, and the actions of the screenshot images are marked schematically to obtain the customer's product attention image dataset. By later recognizing the action images, attention images, and voice information of the customer's attention products, the specific attention points of the customer's attention to a certain product are obtained. For products that have been purchased, the reasons and demand points for purchase are obtained. If not purchased yet, it is found that the detailed features of the customer's attention product do not meet the requirements, the price does not meet the requirements, or the customer's own needs have not been met yet. If a customer does not focus on a certain product when browsing an offline product sales venue, the corresponding video is discarded. When marking, for example, if a customer focuses on a certain ring but looks at the price and does not try it on anymore, it means the customer is considering the price. At this time, only the eye information of the ring that the customer focuses on needs to be recognized. If the customer continues to wear it but is not satisfied with the appearance after passing it on, it means the appearance of the ring is not very satisfactory. At this time, screenshots of the wearing video are needed, usually three to five screenshots. Figure 1 a few major actions.

[0060] Step 4: Aggregate the marked images of each customer to obtain the customer's intended product image. Aggregate the marked images to obtain the customer's intended product image, including those that are being focused on or have already been purchased.

[0061] Step 5: Construct a three - layer learning neural network model, as Figure 2 shown.

[0062] Step 5.1: Set up a perceptron for classifying image nodes. Use a two - layer MLP for image node classification. The activation function of the first layer is ReLU, and a normalization layer and Dropout are used to prevent overfitting. The activation function of the second layer is Softmax, which converts the image node representation into the probability of class prediction. The expression is as follows:

[0063] Z = Dropout(LN(σ(XW 0 )))

[0064]

[0065] where X is the original image node representation, σ() represents the activation function. In the first - layer neural network, ReLU is used, LN() represents layer normalization operation, and Dropout() represents randomly discarding neurons. The two operations work together to prevent overfitting, enabling the model to have better generalization ability, and W 0 ,W 1Represents the trainable parameters of a two-layer neural network, maps the input image node representation to the prediction probabilities of different classes, that is, Y, and obtains the prediction result;

[0066] Step 5.2: Design the first-level contrastive learning module to introduce more edge information to enhance the effect of contrastive learning. The first-level contrastive learning module is an edge-level contrastive learning module, which introduces more edge information through the construction of a multi-hop graph. The multi-hop graph is constructed based on the following rules:

[0067]

[0068] In the construction of the multi-hop graph, if image node a is a neighbor of image node b, and at the same time image node c is a neighbor of image node b, it is considered that image node a is a neighbor of image node c, and an edge is established. More edge information is introduced into the contrastive learning to improve the accuracy of image node prediction;

[0069] The loss function of edge-level contrastive learning is defined as follows:

[0070]

[0071] Among them, B represents the batch size, represents the neighbor set of image node i, and sim(z i , z j ) represents the similarity between image nodes i and j, and τ is the temperature parameter. In the multi-hop graph , if there is an edge between two image nodes, it will make their representations closer, that is, the similarity sim(z i , z j ) increases. Through edge-level contrastive learning, similar image nodes will be more clustered, while dissimilar image nodes will move away from each other. The contrastive learning method can make the distribution of image nodes more compact, thus making the decision boundary of the classifier clearer and finally effectively improving the prediction accuracy;

[0072] Step 5.3: Set the second-level contrastive learning module to generate the perturbed feature representation and further optimize the decision boundary of image node classification. The second-level contrastive learning module is a feature-level contrastive learning module, which perturbs the image node representation by using the randomness of Dropout to strengthen the stability of the model. The specific formula is as follows:

[0073] Z + = Dropout(LN(σ(XW 0 )))

[0074] Among them, z + is the perturbed image node representation generated by Dropout. In addition, Z +Form a contrastive learning sample pair with Z, and use the following loss function to maintain the stability of model training:

[0075]

[0076] in The goal of the loss function is to constrain the original representation Z and the perturbation representation Z + The similarity between the two images enhances the stability and robustness of the model. Even when the image node features are disturbed, the model can still maintain the reliability of the output results.

[0077] Step 5.4: Set the third-level contrastive learning module to introduce a way of disturbing the order of image nodes to achieve the learning of global information without adding additional parameters, effectively controlling the complexity of the model. The third-level contrastive learning module is an image node-level contrastive learning module. The specific process is as follows:

[0078] Z - =shuffle(Z)

[0079] Among them, shuffle() means to disrupt the order of image nodes to generate negative samples Z - In contrastive learning, the constrained image node representation Z and the perturbed image node representation Z - similarity, so as to achieve the goal of keeping the two away from each other;

[0080] The loss function for image node-level contrastive learning is defined as:

[0081]

[0082] There are two cases. The first is image nodes of different categories. When the perturbed image nodes come from different categories, they should obviously be kept away from each other to enhance the distinction between categories. The second is image nodes of the same category. Even if the perturbed image nodes come from the same category, they are still required to be kept away from each other. This is because each image node has its own unique characteristics. By keeping away from other image nodes of the same category, the model can not only capture the commonalities between categories, but also mine the identifying information of the image nodes themselves.

[0083] Step 5.5: Design a total loss function to train the neural network. The total loss function is as follows:

[0084]

[0085] Among them, α, β, and γ are three adjustable hyperparameters. By adjusting these three parameters, we can balance the effects of different loss terms in the model training process and complete the model construction.

[0086] The lightweight two-layer perceptron structure avoids the complex process of embedding the entire graph structure into the neural network, thereby reducing the memory requirement and significantly improving the inference speed. In addition, this design has low requirements for hardware resources and is very suitable for resource-constrained scenarios. In applications with high real-time requirements, the high inference efficiency of the MLP enables it to respond quickly and meet the task requirements. In practical applications, combined with three contrastive learning methods, the MLP not only demonstrates excellent classification performance but also maintains the lightweight and efficiency of the method. Compared with traditional GNN methods, our method effectively reduces the computational resources and time overhead, while enhancing the adaptability of the model in complex graph data. This design that combines efficiency and performance makes our method highly competitive and practical in various application scenarios, providing reliable technical support for solving practical problems.

[0087] Step 6: Input the labeled graph into the third-order learning neural network model to identify the products and product detail information that each customer is interested in. First, it is necessary to extract the features of the nodes from the graph. The purpose of feature learning is to represent the nodes as low-dimensional, real-valued, and dense vector forms for representation and reasoning in the vector space. Taking a customer's interest in a product as the minimum recognition unit, input the labeled graph and voice information related to a product into the third-order learning neural network model. Regard the customer's actions and eye gazes as the recognition image nodes, then perform image recognition, classify the image features, and then predict which aspects of the product the customer's actions are concerned about. Through the model output, each action of the customer is recognized to identify the concerns about the product price, size, function, appearance, color, style, and their own needs.

[0088] When the customer views the product price information but does not purchase, it is determined that the product price is a concern of the customer. Then, when the product price changes, it is used as the main recommendation point for the corresponding customer. When the customer views the product detailed information but does not purchase, it is determined that the product information does not fully meet the customer's needs. When similar products or updated products appear, recommend the similar products or updated products to the customer. After the customer purchases a product, identify the customer information and the information of the purchased product, determine that the corresponding category of customers has a demand for purchasing the corresponding or related products, and recommend them to the corresponding category of customers. When the product that the user views or is interested in has no price and no iterative products, recommend the product that the customer is interested in or views to the customer.

[0089] Step 7: Recommend the products to the corresponding customers from the platforms related to online and offline product sales venues. The merchant will set up corresponding mini-programs or videos to push the products to be recommended to the customer's mobile phone or computer.

[0090] The embodiment of the present invention also provides an electronic device, such as Figure 3As shown, it includes a video acquirer, a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the image recognition-based product recommendation method described in any one of claims 1-7.

[0091] An embodiment of the present invention also provides a computer-readable storage medium with a computer program stored thereon. When the computer program is executed by a processor, it implements the steps in the image recognition-based product recommendation method described in any one of claims 1-7.

[0092] An embodiment of the present invention also provides an application of the image recognition-based product recommendation method, which is applied to online recommendation of all offline product sales, including product recommendations in department stores, clothing stores, electrical appliance stores, and gold stores. As long as it is an offline product sale, it can be transferred to online recommendation, which can achieve the conversion rate of offline sales, or combined with offline collection of customer information for both online recommendation and simultaneous online and offline sales.

[0093] Collect the customer behavior information in the offline sales venue, then perform image recognition to identify the refined focus points of each customer action, and then be able to accurately recommend the required products to the customer. At the same time, according to the refined focus points identified for the customer as price, style, or other refined points, make accurate variable recommendations based on the customer's refined focus points, so that the customer conversion rate is higher and better integration of offline behavior for online recommendation is achieved.

[0094] Matters not described in the present invention are well-known technologies.

[0095] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A commodity recommendation method based on image recognition, characterized in that: The method comprises the following steps: Step 1: The offline product sales place obtains video information about each customer's attention to the product through video; Step 2: Divide the video into several video segments according to offline product sales zones; Step 3: Obtain several labeled images from each video segment; Step 4: Summarize the marked graphs of each customer to obtain the customer's desired product graph; Step 5: Construct a third-order learning neural network model; Step 6: Input the labeled image into the third-order learning neural network model to identify the products that each customer is interested in and the detailed information of the products; Step 7: Recommend products to corresponding customers from platforms related to online and offline product sales venues.

2. The method for recommending products based on image recognition according to claim 1, characterized in that: The specific process of step 1 is: install customer behavior video cameras at offline commodity sales venues to obtain customers' behavior and voice in purchasing commodities at the commodity sales venues, wherein the video cameras perform real-time positioning and tracking according to the number and location of customers, and the obtained video includes the process of customers purchasing commodities and discussing commodities, and then the video and voice are obtained at the same time to obtain a video of the entire process of customers purchasing commodities at the offline commodity sales venues.

3. The method for recommending products based on image recognition according to claim 1, characterized in that: The specific process of step 2 is: taking the customer as the clue, starting when the customer enters the offline commodity sales place, and then ending when the customer leaves the offline commodity sales place, the video is divided into different video segments according to each commodity sales area where the customer is located, and then the title of the video segment is the customer information and the commodity partition information, and the videos in the same commodity partition are aggregated into a group to obtain a video segment group according to the commodity partition, and then the video segment groups of all users are aggregated to obtain a video segment set according to different commodity partitions.

4. The method for recommending products based on image recognition according to claim 1, characterized in that: The specific process of step 3 is as follows: in video tagging, a single item is taken as the smallest unit, and a screenshot segment is taken from the time when the customer pays attention to a certain item to the time when the customer completely leaves the corresponding item. Different video screenshots are taken according to the action, and then the corresponding voice information is read and recognized as text. The screenshot image and voice text information are used as the tag image, and the screenshot image is marked with action indications to obtain the customer's item attention image data set. By subsequently identifying the action image, attention image and voice information of the customer's attention to the item, the specific focus of the customer on a certain item is obtained. For the items that have been purchased, the reasons for purchase and demand points are obtained. If the detailed features of the item that the customer pays attention to do not meet the requirements, the price does not meet the requirements, or the customer's own needs have not been met, the video of the customer not paying attention to a certain item when shopping at an offline item sales venue will be discarded.

5. The commodity recommendation method based on image recognition according to claim 1, characterized in that: The specific process of step 5 is: Step 5.1: Set the perceptron to classify image nodes. Use two layers of MLP to classify image nodes. The activation function of the first layer is ReLU, and the normalization layer and Dropout are used to prevent overfitting. The activation function of the second layer is Softmax. The image node representation is converted into the probability of unclassified prediction. The expression is as follows: Z=Dropout(LN(σ(XW 0 )) Where X is the original image node representation, σ() represents the activation function, and in the first layer of the neural network, ReLU is used, LN() represents the layer normalization operation, and Dropout() represents the random discarding of neurons. The two operations work together to prevent overfitting and make the model have better generalization ability. 0 , W 1 Represents the trainable parameters of the two-layer neural network, which maps the input image node representation to the prediction probability of different categories, that is, Y, to obtain the prediction result; Step 5.2: Design the first-level contrastive learning module to introduce more edge information to enhance the effect of contrastive learning. The first-level contrastive learning module is an edge-level contrastive learning module. It introduces more edge information by constructing a multi-hop graph. The multi-hop graph is constructed based on the following rules: In the construction of a multi-hop graph, if image node a is a neighbor of image node b, and image node c is a neighbor of image node b, image node a is considered to be a neighbor of image node c, and an edge is established. More edge information is introduced into comparative learning, which improves the accuracy of image node prediction. The loss function for edge-level contrastive learning is defined as follows: Where B represents the batch size, represents the neighbor set of image node i, sim(z i , z j ) represents the similarity between image nodes i and j, τ is the temperature parameter, and in the multi-hop graph If there is an edge between two image nodes, their representations will be closer, that is, the similarity sim(z i , z j ) increases. Through edge-level contrastive learning, similar image nodes will be closer together, while dissimilar image nodes will be farther away from each other. The contrastive learning method can make the distribution of image nodes more compact, thereby making the decision boundary of the classifier clearer, and ultimately effectively improving the prediction accuracy. Step 5.3: Set the second-level contrastive learning module to generate perturbed feature representations to further optimize the decision boundary of image node classification. The second-level contrastive learning module is a feature-level contrastive learning module. It perturbs the image node representation by using the randomness of Dropout to enhance the stability of the model. The specific formula is as follows: Z + =Dropout(LN(σ(XW 0 )) Where Z + is the perturbation image node representation generated by Dropout. In addition, Z + Form a contrastive learning sample pair with Z, and use the following loss function to maintain the stability of model training: in The goal of the loss function is to constrain the original representation Z and the perturbation representation Z + The similarity between the two images enhances the stability and robustness of the model. Even when the image node features are disturbed, the model can still maintain the reliability of the output results. Step 5.4: Set the third-level contrastive learning module to introduce a way of disturbing the order of image nodes to achieve the learning of global information without adding additional parameters, effectively controlling the complexity of the model. The third-level contrastive learning module is an image node-level contrastive learning module. The specific process is as follows: Z - =shuffle(Z) Among them, shuffle() means to disrupt the order of image nodes to generate negative samples Z - In contrastive learning, the constrained image node representation Z and the perturbed image node representation Z - similarity, so as to achieve the goal of keeping the two away from each other; The loss function for image node-level contrastive learning is defined as: There are two cases. The first is image nodes of different categories. When the perturbed image nodes come from different categories, they should obviously be kept away from each other to enhance the distinction between categories. The second is image nodes of the same category. Even if the perturbed image nodes come from the same category, they are still required to be kept away from each other. This is because each image node has its own unique characteristics. By keeping away from other image nodes of the same category, the model can not only capture the commonalities between categories, but also mine the identifying information of the image nodes themselves. Step 5.5: Design a total loss function to train the neural network. The total loss function is as follows: l final =l ec +αl edge +βl feature +γl node Among them, α, β, and γ are three adjustable hyperparameters. By adjusting these three parameters, we can balance the effects of different loss terms in the model training process and complete the model construction.

6. The method for recommending products based on image recognition according to claim 1, characterized in that: In step 6, the customer's attention to a product is taken as the minimum recognition unit, and the label map and voice information related to a product are input into the three-order learning neural network model. The customer's actions and eye gaze are used as recognition image nodes, and then image recognition is performed, and the image features are classified. Then, the customer's attention points on the product are predicted, and the customer's attention to the product price, size, function, appearance, color, style and own needs are identified through the model output of each action of the customer; When a customer views the price information of a product but does not purchase it, it is determined that the price of the product is a concern of the customer. When the price of the product changes, it is used as the main recommendation point for the corresponding customer. When a customer views the detailed information of a product but does not purchase it, it is determined that the product information cannot fully satisfy the customer. When similar products or updated products appear, the similar products or updated products are recommended to the customer. If the customer purchases the product, the customer information and the information of the purchased product are identified, and it is determined that the corresponding category of customers has a demand to purchase the corresponding or related products, and the products are recommended to the corresponding category of customers. When the product that the user views or pays attention to does not have a price or iterative product, the product that the customer pays attention to or views will be recommended to the customer.

7. The commodity recommendation method based on image recognition according to claim 1, characterized in that: In step 7, the merchant will set up a corresponding mini-program or video to push the products that need to be recommended to the customer’s mobile phone or computer.

8. An electronic device, characterized in that: The method comprises a video acquirer, a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the commodity recommendation method based on image recognition as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: A computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the steps in the commodity recommendation method based on image recognition as described in any one of claims 1 to 7 are implemented.

10. The application of a commodity recommendation method based on image recognition according to any one of claims 1 to 7, characterized in that: It is used to make online recommendations for all offline merchandise sales, including department store and supermarket merchandise recommendations, clothing store merchandise recommendations, electrical appliance supermarket merchandise recommendations, and gold shop merchandise recommendations.

Images