A self-service cash register method and system based on a knowledge graph

Abstract

This invention discloses a self-service checkout method and system based on a knowledge graph, belonging to the field of self-service checkout technology. The invention obtains candidate products for the goods to be settled through a pre-trained visual recognition model; based on a pre-constructed product knowledge graph, where products are nodes and nodes contain visual similarity edges and confusion intensity edges, the invention calculates the sum of visual similarity and the overall confusion degree of each candidate product as the central node, thereby obtaining the recognition confidence; the feature similarity and recognition confidence of each candidate product are combined to calculate the overall recognition confidence, and the highest confidence is selected as the final recognized product. By introducing a product knowledge graph and comprehensively calculating the recognition confidence of each candidate product using visual similarity edges and confusion intensity edges, the invention effectively corrects the tendency of a single visual model to misjudge similar products, significantly improving the recognition accuracy of specific product types and the efficiency of self-service checkout.

Description

Technical Field

[0001] This invention relates to the field of self-service checkout technology, and in particular to a self-service checkout method and system based on knowledge graphs. Background Technology

[0002] Self-checkout systems are increasingly widely used in the retail sector, playing a particularly important role in scenarios such as fresh food supermarkets and fruit and vegetable retail stores. By integrating functions such as product identification, weighing and pricing, and electronic payment, self-checkout systems can effectively improve checkout efficiency, reduce labor costs, and provide consumers with a more convenient shopping experience. Currently, self-checkout technology is relatively mature, capable of quickly scanning and identifying pre-packaged goods, and automatically obtaining product information based on product barcodes or RFID tags to complete the settlement process.

[0003] However, existing self-checkout technologies face significant limitations when dealing with items like fruits and vegetables, nuts, and fresh meats that require on-site weighing and pricing. Taking fruits and vegetables as an example, the same category often contains multiple varieties with potentially large price differences. For instance, the apple category includes dozens of varieties such as Fuji, Red Delicious, and Golden Delicious, with substantial price variations. Current technology struggles to accurately distinguish these specific varieties using image recognition alone; it often only identifies a few possible varieties within a broad category, then displays those varieties on the interface, requiring manual selection. The system then retrieves the corresponding price information based on the user's selection, combines it with weight data from the weighing sensor, calculates the final price, and processes the transaction. Therefore, the current technology's inability to specifically identify product varieties and its reliance on manual selection by the user increases operational steps, reducing the convenience and efficiency of self-checkout. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention provides a self-service checkout method based on knowledge graphs, comprising the following steps: S1. Based on the product image of the product to be settled, output candidate products with feature similarity greater than or equal to the preset similarity threshold through a pre-trained visual recognition model; S2. Obtain a pre-constructed product knowledge graph, wherein each product is a node and the relationship between each node includes visual similarity edges and confusion intensity edges. Select any candidate product as the center node and the other candidate products as edge nodes. Calculate the sum of visual similarities of the center node based on the visual similarity between the center node and each edge node. Calculate the overall confusion level of the center node based on the confusion intensity between the center node and each edge node. Calculate its recognition confidence based on the sum of visual similarities and the overall confusion level of the center node. S3. Obtain the identification confidence of each candidate product according to the method in step S2, and then calculate the comprehensive identification confidence of each candidate product by combining the corresponding feature similarity. The candidate product with the highest comprehensive identification confidence is taken as the final identified product to be settled. After obtaining the unit price and weight data of the final identified product, the total price of the product to be settled is settled.

[0005] Furthermore, the visual recognition model is an end-to-end deep neural network classifier. Its training phase includes: using a large number of labeled images of each product as initial training samples, training with transfer learning, and after training, the visual recognition model outputs the recognition probability of each product to the input image of the product to be settled, and uses the recognition probability as the feature similarity between each product and the image of the product to be settled.

[0006] Furthermore, the visual similarity edges between nodes are undirected edges, recording the visual similarity between nodes; the confusion intensity edges between nodes are directed edges, recording the confusion intensity of the product corresponding to the starting node for the product corresponding to the pointing node.

[0007] Furthermore, the visual similarity between nodes is obtained through the following steps before constructing the product knowledge graph: Obtain multiple first sample images of the product corresponding to the current node and multiple second sample images of the product corresponding to another node; Using a deep convolutional neural network as a feature extractor, the first feature vector of each first sample image and the second feature vector of each second sample image are extracted respectively. Calculate the cosine similarity between each first feature vector and each second feature vector, and then calculate the average of the cosine similarities to obtain the visual similarity between the corresponding products of the two nodes.

[0008] Furthermore, the method for obtaining the confusion intensity recorded by the confusion intensity edge specifically includes the following steps: Obtain a preset number of product images of different physical objects of the currently verified product as the confusion strength verification set of the product, wherein the preset number is the same as the preset number of verification recognition attempts; The product images in the confusion strength verification set are identified one by one by the visual recognition model to obtain verification candidate products with feature similarity greater than or equal to a preset similarity threshold. The verification candidate products do not include the current verification product. The ratio of the number of times each verification candidate product appears to the preset number of verification identifications is used as the confusion strength of the current verification product for the corresponding verification candidate product. Record each confusion intensity on the edge pointing to the corresponding candidate product of the node corresponding to the current verified product.

[0009] Furthermore, the step of calculating the overall confusion degree of the central node based on the confusion intensity between the central node and each edge node specifically includes: The first sum of confusion intensity of the central node is calculated based on the confusion intensity edges between each edge node as the starting node and the central node. The second sum of confusion intensity of the central node is calculated based on the confusion intensity edges between the central node as the starting node and each edge node. The comprehensive confusion degree of the central node is obtained by subtracting the first sum of confusion intensity from the second sum of confusion intensity.

[0010] Furthermore, the calculation of the recognition confidence is specifically as follows: based on the preset similarity weight and the preset confusion weight, the sum of the visual similarity of the central nodes and the sum of the comprehensive confusion intensity are weighted and calculated to obtain the recognition confidence. The calculation of the comprehensive identification confidence score is specifically as follows: the feature similarity and identification confidence scores corresponding to the candidate products are weighted and calculated according to preset weights to obtain the comprehensive identification confidence score.

[0011] Furthermore, after selecting the candidate product with the highest overall confidence level as the final identified product to be settled in step S3, the following steps are also included: Determine whether the finally identified product matches the candidate product with the highest feature similarity output by the visual recognition model: If they match, then directly update the confusion intensity recorded on the confusion intensity edge between the node corresponding to the finally identified product and the nodes corresponding to other candidate products in the knowledge graph; If there is a discrepancy, the product image of the product to be settled is labeled as the final identified product and placed into the sample pool corresponding to the final identified product. At the same time, the confusion intensity recorded on the confusion intensity edge between the node corresponding to the final identified product and the nodes corresponding to other candidate products in the knowledge graph is updated. If the number of samples in the sample pool is greater than or equal to the preset number of samples, the visual recognition model is updated using the samples in the sample pool.

[0012] Furthermore, the confusion intensity recorded on the confusion intensity edge between the node corresponding to the finally identified product and the nodes corresponding to other candidate products in the updated knowledge graph is specifically as follows: Obtain the confusion strength edge between the node corresponding to the finally identified product and the nodes corresponding to other candidate products, and record it as the outward confusion strength edge; obtain the number of times each other candidate product appears as a verification candidate product when the finally identified product is the current verification product, and record it as the number of misjudgments; In the calculation of the confusion intensity of the corresponding outgoing confusion intensity edge, the latest misjudgment count of other candidate products corresponding to the outgoing confusion intensity edge is increased by one and the latest verification and recognition count of the finally identified product is increased by one. The ratio of the updated misjudgment count to the updated verification and recognition count is used as the confusion intensity of the corresponding outgoing confusion intensity edge. The initial value of the verification and recognition count is the preset verification and recognition count.

[0013] This invention also provides a knowledge graph-based self-service checkout system, comprising: The identification module outputs candidate products with feature similarity greater than or equal to a preset similarity threshold based on the product image of the product to be settled through a pre-trained visual recognition model. The confidence calculation module obtains a pre-constructed product knowledge graph, in which each product is used as a node, and the relationship edges between each node include visual similarity edges and confusion intensity edges. Select any candidate product as the center node and the other candidate products as edge nodes. Calculate the sum of visual similarities of the center node based on the visual similarity between the center node and each edge node. Calculate the overall confusion level of the center node based on the confusion intensity between the center node and each edge node. Calculate its recognition confidence based on the sum of visual similarities and the overall confusion level of the center node. The settlement module obtains the identification confidence of each candidate product, and then calculates the comprehensive identification confidence of each candidate product by combining the corresponding feature similarity. The candidate product with the highest comprehensive identification confidence is selected as the final identified product to be settled. After obtaining the unit price of the final identified product and the weighing data to be settled, the total price of the product to be settled is settled.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention introduces a product knowledge graph, calculates the recognition confidence of each candidate product by combining visual similarity edges and confusion intensity edges, and then weights and fuses it with the feature similarity output by the visual recognition model. This can effectively correct the tendency of a single visual model to misjudge similar products, significantly improve the recognition accuracy of specific product varieties, and automatically output the candidate product with the highest comprehensive recognition confidence as the final recognition result, thereby improving the convenience and intelligence of self-checkout. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a knowledge graph-based self-service checkout method according to the present invention; Figure 2 A flowchart illustrating the process of obtaining visual similarity between nodes in this invention; Figure 3 This is a flowchart of the method for obtaining the obfuscation intensity in this invention; Figure 4 This is a flowchart illustrating the update process of the confusion intensity recorded on the confusion intensity edge in this invention. Figure 5 This is a structural block diagram of a knowledge graph-based self-service checkout system according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0020] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0021] This invention aims to address the technical problem of existing self-checkout systems struggling to accurately distinguish between visually similar product types (such as different varieties of apples, nuts, and fresh meat) when relying solely on a single visual recognition model, thus reducing checkout efficiency and user experience. To overcome this bottleneck, this invention constructs a product knowledge graph containing product nodes, visual similarity edges, and confusion intensity edges, and designs a multi-dimensional confidence correction mechanism based on this graph: Visual similarity edges in the knowledge graph capture the "centrality" tendency of candidate products in the feature space, giving higher confidence to products located at the visual cluster center; confusion intensity edges effectively characterize the asymmetric misjudgment probability of the model among products, correcting systematic misjudgments caused by training bias in a single model; finally, the graph information and model output are used for joint decision-making, significantly improving the accuracy of automatic product identification for similar products, thereby comprehensively enhancing the intelligence level and settlement efficiency of self-checkout.

[0022] Example 1 See Figure 1 As shown, the self-service checkout method based on knowledge graphs provided by this invention specifically includes the following steps: S1. Based on the product image of the product to be settled, output candidate products with feature similarity greater than or equal to the preset similarity threshold through a pre-trained visual recognition model; S2. Obtain a pre-constructed product knowledge graph, wherein each product is a node and the relationship between each node includes visual similarity edges and confusion intensity edges. Select any candidate product as the center node and the other candidate products as edge nodes. Calculate the sum of visual similarities of the center node based on the visual similarity between the center node and each edge node. Calculate the overall confusion level of the center node based on the confusion intensity between the center node and each edge node. Calculate its recognition confidence based on the sum of visual similarities and the overall confusion level of the center node. S3. Obtain the identification confidence of each candidate product according to the method in step S2, and then calculate the comprehensive identification confidence of each candidate product by combining the corresponding feature similarity. The candidate product with the highest comprehensive identification confidence is taken as the final identified product to be settled. After obtaining the unit price and weight data of the final identified product, the total price of the product to be settled is settled.

[0023] When a visual recognition model outputs multiple candidate items, these candidate items are very close to each other in the visual feature space. In other words, they form a "visual similarity cluster." For example, the visual feature vectors of the three apple varieties—Fuji, Gala, and Golden Delicious—are close to each other in space. When recognizing a real Fuji apple, its visual feature vector will fall within the probability distribution area of ​​the "Fuji" variety in the feature space. Since Fuji has a high visual similarity to Gala and Golden Delicious, its feature vector may also be close to the distribution area of ​​Gala and Golden Delicious. Therefore, when matching, the visual recognition model will output a high feature similarity for Fuji, Gala, and Golden Delicious simultaneously. Thus, relying solely on the feature similarity output by the visual recognition model is insufficient to determine an accurate recognition result.

[0024] In real-world visual feature spaces, different varieties of the same product category (such as apples) often exhibit clustering structures, with some varieties located at the cluster center and others distributed around them. The average distance between the feature vectors of the central variety and the feature vectors of other varieties is smaller (i.e., the visual similarity between the central variety and other varieties is higher). Therefore, when a visual recognition model outputs multiple candidate products simultaneously, the product that serves as the central variety among these candidates is often the most likely to be the actual product, and its "visual similarity weight sum" is often higher than that of other candidate varieties. Thus, a higher visual similarity weight sum indicates that the candidate variety is closer to the "visual feature center" in the current candidate set, and the greater the likelihood that it is the actual product.

[0025] In visual recognition, feature vectors of similar products (such as different apple varieties) often form clusters in space. Products located at the center of a cluster have a higher sum of visual similarity to surrounding products, and are therefore more likely to be the real products. However, relying solely on visual similarity ignores the model's tendency to misclassify—for example, the model is more likely to misclassify A as B, and vice versa. This solution introduces confusion intensity edges from a knowledge graph to characterize this asymmetric misclassification probability. By calculating the combined degree of each candidate product's "misclassification of others" and "being misclassified by others," the bias of visual similarity is corrected. Finally, the feature similarity output by the visual recognition model is weighted and fused with the recognition confidence based on the knowledge graph to comprehensively determine the real products.

[0026] In step S1, the visual recognition model is an end-to-end deep neural network classifier. Its training phase includes: using a large number of labeled images of each product as initial training samples, training with transfer learning, and after training, the visual recognition model outputs the recognition probability of each product to the input image of the product to be settled, and uses the recognition probability as the feature similarity between each product and the image of the product to be settled.

[0027] Step S1 specifically includes the following steps: Model inference: The product image of the product to be settled is scaled to the input size required by the visual recognition model, standardized, and then input into the visual recognition model. The visual recognition model performs forward propagation and finally outputs a probability distribution vector P=[p1,p2,...,pc], where p is the recognition probability of the corresponding product and also the feature similarity between the corresponding product and the product to be settled, c is the number of product types, the value range of each recognition probability p is [0,1], and the sum of all recognition probabilities p is 1; Candidate product selection: Obtain a preset similarity threshold and select products with feature similarity greater than the preset similarity threshold as candidate products.

[0028] In some embodiments, the preset similarity threshold can be obtained through the following steps: The products identified by the visual recognition model are sorted from highest to lowest based on feature similarity. In the sequence, calculate the difference between the feature similarity of the first item and the feature similarity of the other items one by one; The query finds the smallest difference that is greater than or equal to the preset difference threshold, and uses the feature similarity of the last feature after the last smallest difference as the preset similarity threshold for this identification.

[0029] For example, in a single identification process, the identified products are A, B, C, D, and E, with feature similarities of 0.35, 0.3, 0.25, 0.05, and 0.05, respectively. The difference sequence is 0.05, 0.05, 0.2, and 0.2. If the preset difference threshold is set to 0.06, the minimum difference is 0.05. The last minimum difference is the second 0.05 in the difference sequence, so the next feature similarity is 0.25. Therefore, if the preset similarity threshold for this identification is set to 0.25, then the candidate products in this identification are A, B, and C.

[0030] In some embodiments, if the visual recognition model does not output any candidate products, the user is prompted to reposition the product for recognition. If no candidate products are output after a preset number of consecutive empty recognition attempts, an error message is sent to the administrator.

[0031] In step S2, the visual similarity edges between nodes are undirected edges, and the visual similarity edges record the visual similarity between nodes; the confusion intensity edges between nodes are directed edges, and the confusion intensity edges record the confusion intensity of the product corresponding to the starting node for the product corresponding to the pointing node.

[0032] It should be noted that the confusion strength edge between nodes is divided into two parts, namely, the starting node and the pointing node are different nodes. For example, for nodes A and B, the confusion strength edge between the two nodes includes node A as the starting node pointing to node B (in this case, node B is the pointing node), and node B as the starting node pointing to node A (in this case, node A is the pointing node). The confusion strength recorded by the confusion strength edge with node A as the starting node pointing to node B is the confusion strength of the product corresponding to node A to the product corresponding to node B.

[0033] When visual recognition models identify product varieties, their misidentification is often asymmetrical. For example, "Fuji apples" and "Gala apples" have similar visual features. Because some visual features of Fuji apples (such as a redder base color and striped distribution) overlap with typical Gala apple features, while Gala apples' unique features (such as smaller size and brighter yellow) are more easily captured by the model, the model may be more inclined to misidentify Fuji apples as Gala apples, but conversely, less likely to misidentify Gala apples as Fuji apples. Therefore, in this scheme, the confusion intensity edges between nodes are directed edges and divided into two to represent the confusion intensity of products corresponding to different starting nodes for products corresponding to different pointing nodes.

[0034] See Figure 2 As shown, in step S2, the visual similarity between nodes is obtained through the following steps before constructing the product knowledge graph: Obtain multiple first sample images of the product corresponding to the current node and multiple second sample images of the product corresponding to another node; Using a deep convolutional neural network as a feature extractor, the first feature vector of each first sample image and the second feature vector of each second sample image are extracted respectively. Calculate the cosine similarity between each first feature vector and each second feature vector, and then calculate the average of the cosine similarities to obtain the visual similarity between the corresponding products of the two nodes.

[0035] The deep convolutional neural network is pre-trained on a large image dataset (such as ImageNet).

[0036] In the high-dimensional feature space extracted by deep convolutional neural networks (CNNs), the magnitude of feature vectors is easily affected by external factors during image acquisition, such as light intensity, color temperature changes, shooting angle, and distance differences. The CNN feature vectors of the same product (e.g., two Fuji apples photographed under different lighting conditions) typically maintain a high degree of consistency in direction, but their magnitudes can fluctuate significantly due to the aforementioned noise. Euclidean distance, which measures both directional and magnitude differences, can lead to the same product having a greater distance from its own class center in the feature space than from other products due to different shooting conditions. Cosine similarity, on the other hand, focuses only on the angle (direction) between feature vectors and possesses a "normalization" property for vector magnitudes. This effectively eliminates the interference of gain-type noise such as lighting and contrast, allowing visual similarity calculations to focus more on semantic-level differences in shape, texture, and color structure. Therefore, compared to Euclidean distance, this scheme uses cosine similarity to measure the directional consistency of two feature vectors, which is unaffected by vector magnitudes and is suitable for similarity comparisons in high-dimensional feature spaces. Features extracted by deep convolutional neural networks have a semantic hierarchy and can capture visual attributes such as shape, texture, and color.

[0037] In some embodiments, the deep convolutional neural network in step S2 is selected to be the ResNet series. In image retrieval tasks like those in this invention, ResNet is widely used as the first choice for feature extractors because its deep structure can effectively extract semantic-level features. Furthermore, the scheme uses cosine similarity to measure the distance between feature vectors, focusing on the directional consistency of feature vectors. The residual structure of ResNet encourages features to move along the negative gradient direction of the loss function during training for iterative optimization. This makes the features extracted by ResNet have better clustering characteristics on the hypersphere, which is more in line with the measurement premise of cosine similarity.

[0038] See Figure 3 As shown, in step S2, the method for obtaining the confusion intensity recorded by the confusion intensity edge specifically includes the following steps: Obtain a preset number of product images of different physical objects of the currently verified product as the confusion strength verification set of the product, wherein the preset number is the same as the preset number of verification recognition attempts; The product images in the confusion strength verification set are identified one by one by the visual recognition model to obtain verification candidate products with feature similarity greater than or equal to a preset similarity threshold. The verification candidate products do not include the current verification product. The ratio of the number of times each verification candidate product appears to the preset number of verification identifications is used as the confusion strength of the current verification product for the corresponding verification candidate product. Record each confusion intensity on the edge pointing to the corresponding candidate product of the node corresponding to the current verified product.

[0039] For example, in obtaining the confusion intensity of product A among various products: images of different physical products of product A are obtained (e.g., Fuji apples; if the preset number of images is 3, then 3 different images of Fuji apples are obtained. It should be noted that in actual verification, the base number of the preset number will be relatively large; here, the preset number is set to a small base of 3 for ease of explanation). The images of each product A are then identified through the visual recognition model. For example, the verification candidate products obtained by identifying image 1 include product B and product C, the verification candidate products obtained by identifying image 2 include product B, product C, and product D, and the verification candidate products obtained by identifying image 1 include product B and product D. Therefore, the confusion intensity of product A among product B is 1, reflecting that the visual recognition model is extremely prone to misidentifying product A as product B when identifying product A.

[0040] Confusion strength is essentially an estimate of the conditional probability that a visual recognition model misclassifies an image of one product as another in actual testing. By conducting multiple independent tests (with different images of the same product), the frequency of misclassification is statistically analyzed. According to the law of large numbers, the frequency approaches the probability. Since the model may produce different misclassification results for different images of the same product (for example, a Fuji apple might be misclassified as a Gala apple due to its obvious stripes, while another might be correctly identified due to its lighter color), this approach uses multiple images of different products to cover variations in product appearance, avoids overfitting to the features of a single product, and obtains a more representative confusion strength.

[0041] In step S2, the calculation of the overall confusion degree of the central node based on the confusion intensity between the central node and each edge node specifically includes: The first sum of confusion intensity of the central node is calculated based on the confusion intensity edges between each edge node as the starting node and the central node. The second sum of confusion intensity of the central node is calculated based on the confusion intensity edges between the central node as the starting node and each edge node. The comprehensive confusion degree of the central node is obtained by subtracting the first sum of confusion intensity from the second sum of confusion intensity.

[0042] The first sum of confusion intensity of the central node is the sum of the confusion intensity recorded by the confusion intensity edges between each edge node as the starting node and the central node; the first sum of confusion intensity reflects the degree to which the candidate product represented by the current central node is easily misjudged by other candidate products within the candidate product range.

[0043] The second confusion intensity summation of the central node is the sum of the confusion intensity recorded by the confusion intensity edges between the central node as the starting node and each edge node; the second confusion intensity summation reflects the degree to which the candidate product represented by the current central node is easily misclassified as other candidate products within the candidate product range.

[0044] Subtracting the sum of the first and second confusion intensities from the sum of the second confusion intensities yields the overall confusion degree of the central node. This reflects whether the candidate product represented by the current central node is easily misjudged or misjudged as another candidate product within the current range of candidate products. A positive value indicates that it is easily misjudged as another candidate product, while a negative value indicates that it is easily misjudged as another candidate product. In other words, the greater the overall confusion degree, the higher the probability that the candidate product represented by the current central node is the true identified product.

[0045] In step S2, the recognition confidence is calculated based on the sum of visual similarity of the central nodes and the sum of comprehensive confusion intensity. Specifically, the recognition confidence is calculated by weighting the sum of visual similarity of the central nodes and the sum of comprehensive confusion intensity according to the preset similarity weight and preset confusion weight.

[0046] In step S3, the calculation method for the comprehensive identification confidence of candidate items is as follows: The comprehensive recognition confidence score is obtained by weighting and calculating the feature similarity and recognition confidence scores of the candidate products according to preset weights.

[0047] In some embodiments, after step S3 identifies the candidate product with the highest overall confidence level as the final identified product to be settled, it further includes the following steps: Determine whether the finally identified product matches the candidate product with the highest feature similarity output by the visual recognition model: If they match, then directly update the confusion intensity recorded on the confusion intensity edge between the node corresponding to the finally identified product and the nodes corresponding to other candidate products in the knowledge graph; If there is a discrepancy, the product image of the product to be settled is labeled as the final identified product and placed into the sample pool corresponding to the final identified product. At the same time, the confusion intensity recorded on the confusion intensity edge between the node corresponding to the final identified product and the nodes corresponding to other candidate products in the knowledge graph is updated. If the number of samples in the sample pool is greater than or equal to the preset number of samples, the visual recognition model is updated using the samples in the sample pool.

[0048] See Figure 4 As shown, the confusion intensity recorded on the confusion intensity edge between the node corresponding to the finally identified product and the nodes corresponding to other candidate products in the updated knowledge graph is specifically as follows: Obtain the confusion strength edge between the node corresponding to the finally identified product and the nodes corresponding to other candidate products, and record it as the outward confusion strength edge; obtain the number of times each other candidate product appears as a verification candidate product when the finally identified product is the current verification product, and record it as the number of misjudgments; In the calculation of the confusion intensity of the corresponding outgoing confusion intensity edge, the latest misjudgment count of other candidate products corresponding to the outgoing confusion intensity edge is increased by one and the latest verification and recognition count of the finally identified product is increased by one. The ratio of the updated misjudgment count to the updated verification and recognition count is used as the confusion intensity of the corresponding outgoing confusion intensity edge. The initial value of the verification and recognition count is the preset verification and recognition count.

[0049] When the model's recognition result differs from the result corrected by the knowledge graph, it indicates that the current visual recognition model has a bias towards the image of the product, or that the confusion strength in the knowledge graph is outdated. In this case, the system stores the image as a correctly labeled sample in the sample pool and immediately updates the confusion strength, as this checkout is essentially a "validation experiment." When the corresponding sample pool accumulates sufficiently, it triggers incremental learning of the visual recognition model, allowing the model to adapt to the new image distribution. Even if the results are consistent, the confusion strength must still be updated, because each correct recognition provides new validation data.

[0050] In some embodiments, during the acquisition of the final identified commodity's pending weighing data, the system does not acquire weighing data or perform any weighing-related settlement operations before the commodity type is determined. Only after the final identified commodity is determined does the system trigger the acquisition of weighing data: after determining the final identified commodity, the system sends a weighing data reading instruction to the weighing device; the weighing device responds to the reading instruction and returns the weight data of the commodity on the current weighing pan; the system calculates the total price based on the unit price of the final identified commodity and the acquired weight data, and completes the settlement.

[0051] Example 2 See Figure 5 As shown, the present invention also provides a self-service checkout system based on a knowledge graph, specifically including: The system includes an identification module, a confidence calculation module, and a settlement module, as well as a central processing unit that communicates with the aforementioned modules.

[0052] The identification module outputs candidate products with feature similarity greater than or equal to a preset similarity threshold based on the product image of the product to be settled through a pre-trained visual recognition model. The confidence calculation module obtains a pre-constructed product knowledge graph, in which each product is used as a node, and the relationship edges between each node include visual similarity edges and confusion intensity edges. Select any candidate product as the center node and the other candidate products as edge nodes. Calculate the sum of visual similarities of the center node based on the visual similarity between the center node and each edge node. Calculate the overall confusion level of the center node based on the confusion intensity between the center node and each edge node. Calculate its recognition confidence based on the sum of visual similarities and the overall confusion level of the center node. The settlement module obtains the identification confidence of each candidate product, and then calculates the comprehensive identification confidence of each candidate product by combining the corresponding feature similarity. The candidate product with the highest comprehensive identification confidence is selected as the final identified product to be settled. After obtaining the unit price of the final identified product and the weighing data to be settled, the total price of the product to be settled is settled.

[0053] The recognition module specifically acquires images of the goods to be settled through a photographic device, and then transmits them to the visual recognition model for recognition.

[0054] The unit price of the goods is pre-entered into the unit price database of the self-service checkout system, and the weighing data to be settled is the real-time weighing data obtained by the weighing device of the goods to be settled.

[0055] Example 3 The present invention also provides an electronic device, including: a processor, a transmitting device, an input device, an output device, and a memory. The processor may be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory may be implemented using a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), and is used to store computer program code. The computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes a method as described in any of the above possible implementation methods.

[0056] Example 4 The present invention also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor of an electronic device, cause the processor to perform a method as described in any of the above possible implementations.

[0057] The beneficial effects of this invention are as follows: This invention introduces a product knowledge graph, calculates the recognition confidence of each candidate product by combining visual similarity edges and confusion intensity edges, and then weights and fuses it with the feature similarity output by the visual recognition model. This can effectively correct the tendency of a single visual model to misjudge similar products, significantly improve the recognition accuracy of specific product varieties, and automatically output the candidate product with the highest comprehensive recognition confidence as the final recognition result, thereby improving the convenience and intelligence of self-checkout.

[0058] In the description of this specification, the references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0059] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0060] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A self-service checkout method based on knowledge graphs, characterized in that, Includes the following steps: S1. Based on the product image of the product to be settled, output candidate products with feature similarity greater than or equal to the preset similarity threshold through a pre-trained visual recognition model; S2. Obtain a pre-constructed product knowledge graph, wherein each product is a node and the relationship between each node includes visual similarity edges and confusion intensity edges. Select any candidate product as the center node and the other candidate products as edge nodes. Calculate the sum of visual similarities of the center node based on the visual similarity between the center node and each edge node. Calculate the overall confusion level of the center node based on the confusion intensity between the center node and each edge node. Calculate its recognition confidence based on the sum of visual similarities and the overall confusion level of the center node. S3. Obtain the identification confidence of each candidate product according to the method in step S2, and then calculate the comprehensive identification confidence of each candidate product by combining the corresponding feature similarity. The candidate product with the highest comprehensive identification confidence is taken as the final identified product to be settled. After obtaining the unit price and weight data of the final identified product, the total price of the product to be settled is settled.

2. The self-service checkout method based on knowledge graphs according to claim 1, characterized in that, The visual recognition model is an end-to-end deep neural network classifier. Its training phase includes: using a large number of labeled images of each product as initial training samples, training with transfer learning, and after training, the visual recognition model outputs the recognition probability of each product to the input image of the product to be settled, and uses the recognition probability as the feature similarity between each product and the image of the product to be settled.

3. The self-service checkout method based on knowledge graphs according to claim 1, characterized in that, The visual similarity edges between nodes are undirected edges, recording the visual similarity between nodes; the confusion intensity edges between nodes are directed edges, recording the confusion intensity of the product corresponding to the starting node with respect to the product corresponding to the pointing node.

4. The self-service checkout method based on knowledge graphs according to claim 3, characterized in that, The visual similarity between nodes is obtained through the following steps before constructing the product knowledge graph: Obtain multiple first sample images of the product corresponding to the current node and multiple second sample images of the product corresponding to another node; Using a deep convolutional neural network as a feature extractor, the first feature vector of each first sample image and the second feature vector of each second sample image are extracted respectively. Calculate the cosine similarity between each first feature vector and each second feature vector, and then calculate the average of the cosine similarities to obtain the visual similarity between the corresponding products of the two nodes.

5. The self-service checkout method based on knowledge graphs according to claim 3, characterized in that, The method for obtaining the confusion intensity recorded by the confusion intensity edge specifically includes the following steps: Obtain a preset number of product images of different physical objects of the currently verified product as the confusion strength verification set of the product, wherein the preset number is the same as the preset number of verification recognition attempts; The product images in the confusion strength verification set are identified one by one by the visual recognition model to obtain verification candidate products with feature similarity greater than or equal to a preset similarity threshold. The verification candidate products do not include the current verification product. The ratio of the number of times each verification candidate product appears to the preset number of verification identifications is used as the confusion strength of the current verification product for the corresponding verification candidate product. Record each confusion intensity on the edge pointing to the corresponding candidate product of the node corresponding to the current verified product.

6. The self-service checkout method based on knowledge graphs according to claim 3, characterized in that, The step of calculating the overall confusion level of the central node based on the confusion intensity between the central node and each edge node specifically includes: The first sum of confusion intensity of the central node is calculated based on the confusion intensity edges between each edge node as the starting node and the central node. The second sum of confusion intensity of the central node is calculated based on the confusion intensity edges between the central node as the starting node and each edge node. The comprehensive confusion degree of the central node is obtained by subtracting the first sum of confusion intensity from the second sum of confusion intensity.

7. The self-service checkout method based on knowledge graphs according to claim 1, characterized in that, The calculation of the recognition confidence is specifically as follows: based on the preset similarity weight and preset confusion weight, the sum of the visual similarity of the central nodes and the sum of the comprehensive confusion intensity are weighted and calculated to obtain the recognition confidence. The calculation of the comprehensive identification confidence score is specifically as follows: the feature similarity and identification confidence scores corresponding to the candidate products are weighted and calculated according to preset weights to obtain the comprehensive identification confidence score.

8. The self-service checkout method based on knowledge graphs according to claim 5, characterized in that, After selecting the candidate product with the highest overall confidence level as the final identified product to be settled, step S3 also includes the following steps: Determine whether the finally identified product matches the candidate product with the highest feature similarity output by the visual recognition model: If they match, then directly update the confusion intensity recorded on the confusion intensity edge between the node corresponding to the finally identified product and the nodes corresponding to other candidate products in the knowledge graph; If there is a discrepancy, the product image of the product to be settled is labeled as the final identified product and placed into the sample pool corresponding to the final identified product. At the same time, the confusion intensity recorded on the confusion intensity edge between the node corresponding to the final identified product and the nodes corresponding to other candidate products in the knowledge graph is updated. If the number of samples in the sample pool is greater than or equal to the preset number of samples, the visual recognition model is updated using the samples in the sample pool.

9. The self-service checkout method based on knowledge graphs according to claim 8, characterized in that, The confusion intensity recorded on the confusion intensity edge between the node corresponding to the finally identified product and the nodes corresponding to other candidate products in the updated knowledge graph is specifically as follows: Obtain the confusion strength edge between the node corresponding to the finally identified product and the nodes corresponding to other candidate products, and record it as the outward confusion strength edge; obtain the number of times each other candidate product appears as a verification candidate product when the finally identified product is the current verification product, and record it as the number of misjudgments; In the calculation of the confusion intensity of the corresponding outgoing confusion intensity edge, the latest misjudgment count of other candidate products corresponding to the outgoing confusion intensity edge is increased by one and the latest verification and recognition count of the finally identified product is increased by one. The ratio of the updated misjudgment count to the updated verification and recognition count is used as the confusion intensity of the corresponding outgoing confusion intensity edge. The initial value of the verification and recognition count is the preset verification and recognition count.

10. A knowledge graph-based self-checkout system, applied to the knowledge graph-based self-checkout method described in any one of claims 1 to 9, characterized in that, include: The identification module outputs candidate products with feature similarity greater than or equal to a preset similarity threshold based on the product image of the product to be settled through a pre-trained visual recognition model. The confidence calculation module obtains a pre-constructed product knowledge graph, in which each product is used as a node, and the relationship edges between each node include visual similarity edges and confusion intensity edges. Select any candidate product as the center node and the other candidate products as edge nodes. Calculate the sum of visual similarities of the center node based on the visual similarity between the center node and each edge node. Calculate the overall confusion level of the center node based on the confusion intensity between the center node and each edge node. Calculate its recognition confidence based on the sum of visual similarities and the overall confusion level of the center node. The settlement module obtains the identification confidence of each candidate product, and then calculates the comprehensive identification confidence of each candidate product by combining the corresponding feature similarity. The candidate product with the highest comprehensive identification confidence is selected as the final identified product to be settled. After obtaining the unit price of the final identified product and the weighing data to be settled, the total price of the product to be settled is settled.

Images