Shopping cart cluster environmental information perception method, device and system

Abstract

The present invention discloses a method, device and system for sensing environmental information of a shopping cart cluster. The method comprises: after receiving an information collection instruction, obtaining a video stream and simultaneously collecting its own environmental information; the information collection instruction is sent by an environmental perception server after obtaining the environmental information required to be collected for each shopping cart according to the accumulated shopping environment information and the position of each shopping cart; inputting the information collection instruction and the video stream into a target detection model to obtain the commodity information of the detected commodity; summarizing its own environmental information and the detected commodity information to obtain the shopping environment information; the target detection model comprises a selection mechanism decision module for determining an anchor point frame set and a feature vector corresponding to an input image; a pre-trained model for image-text comparison is used to obtain at least one text feature and multiple image features according to the anchor point frame set and the feature vector and the information collection instruction to obtain the detected commodity. The present invention can realize open target detection.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a method, device and system for sensing environmental information of a shopping cart cluster. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] In supermarkets, shopping cart environmental perception has become a new research field. Environmental perception often requires the use of artificial intelligence models. As a classic artificial intelligence model, the language large model has been widely used in many fields. However, as an edge device, the computing power of the shopping cart is limited, and it is difficult to deploy a complex language large model, which makes the shopping cart unable to have strong intelligence; and the lack of background information and contextual content also makes the language large model unable to make accurate answers and responses to shopping scenarios. Although the small model used for environmental perception can be deployed in the shopping cart, it can usually only detect targets of fixed categories and lacks focus on the goods that need to be detected. When a single shopping cart works independently, a large number of false detection results will be generated due to the limited capabilities of the small model. Even if there are multiple shopping carts in the same environment, the perception of the environment cannot be verified by each other. Therefore, a system that combines end-cloud collaboration with large models and open target detection is imperative. However, there is currently a lack of a system that improves the environmental perception capability of the small model while obtaining the generalization and emergence capabilities of the large model. Summary of the invention

[0004] The embodiment of the present invention provides a method for sensing environmental information of a shopping cart cluster, which is applied to a shopping cart and can detect any non-predefined target commodity, thereby realizing open target detection and having strong environmental sensing capability. The method includes:

[0005] After receiving the information collection instruction, the video stream is obtained and the environment information of the shopping cart is collected at the same time; the information collection instruction is sent by the environment perception server to each shopping cart after the environment information that needs to be collected for each shopping cart in the shopping cart cluster is obtained based on the accumulated shopping environment information and the position of each shopping cart, and the information collection instruction includes the environment information that needs to be collected by the shopping cart;

[0006] Input the information collection instruction and the video stream into the target detection model to obtain the commodity information of the commodity detected from the video stream;

[0007] Aggregate the own environment information and the detected commodity information to obtain the shopping environment information of the shopping cart;

[0008] The target detection model includes:

[0009] A selection mechanism decision module is used to determine a set of anchor boxes and corresponding feature vectors corresponding to an input image corresponding to a video stream;

[0010] A pre-trained model for image-text comparison is used to obtain at least one text feature and multiple image features based on a set of anchor boxes and corresponding feature vectors and information collection instructions; calculate confidence based on the at least one text feature and multiple image features; determine a product with a confidence greater than a preset confidence threshold as a detected product, and output product information of the detected product.

[0011] The embodiment of the present invention provides a shopping cart cluster environment information sensing device, which is applied to a shopping cart and can detect any non-predefined target commodity, thereby realizing open target detection and having strong environment sensing capability. The device includes:

[0012] An information collection module, which is used to obtain the video stream and collect its own environment information after receiving an information collection instruction. The information collection instruction is sent by the environment perception server to each shopping cart after the environment perception server obtains the environment information that needs to be collected for each shopping cart in the shopping cart cluster based on the accumulated shopping environment information and the position of each shopping cart. The information collection instruction includes the environment information that needs to be collected by the shopping cart;

[0013] A commodity acquisition module, used to input the information collection instruction and the video stream into the target detection model to obtain commodity information of the commodity detected from the video stream;

[0014] A shopping environment information acquisition module is used to aggregate its own environment information and the commodity information of the detected commodities to obtain the shopping environment information of the shopping cart;

[0015] The target detection model includes:

[0016] A selection mechanism decision module is used to determine a set of anchor boxes and corresponding feature vectors corresponding to an input image corresponding to a video stream;

[0017] A pre-trained model for image-text comparison is used to obtain at least one text feature and multiple image features based on a set of anchor boxes and corresponding feature vectors and information collection instructions; calculate confidence based on the at least one text feature and multiple image features; determine a product with a confidence greater than a preset confidence threshold as a detected product, and output product information of the detected product.

[0018] The embodiment of the present invention provides a shopping cart cluster environment information perception system, which can detect any non-predefined target commodity, thereby realizing open target detection and having strong environment perception capability. The system includes: an environment perception server and a shopping cart;

[0019] Context-aware servers for:

[0020] According to the accumulated shopping environment information and the location of each shopping cart, the environment information that needs to be collected for each shopping cart in the shopping cart cluster is obtained;

[0021] Sending an information collection instruction to each shopping cart, wherein the information collection instruction includes environmental information that the shopping cart needs to collect;

[0022] The shopping cart is used for:

[0023] After receiving the information collection instruction, it obtains the video stream and collects its own environmental information at the same time;

[0024] Input the information collection instruction and the video stream into the target detection model to obtain the commodity information of the commodity detected from the video stream;

[0025] Aggregate the own environment information and the detected commodity information to obtain the shopping environment information of the shopping cart;

[0026] The target detection model includes:

[0027] A selection mechanism decision module is used to determine a set of anchor boxes and corresponding feature vectors corresponding to an input image corresponding to a video stream;

[0028] A pre-trained model for image-text comparison is used to obtain at least one text feature and multiple image features based on a set of anchor boxes and corresponding feature vectors and information collection instructions; calculate confidence based on the at least one text feature and multiple image features; determine a product with a confidence greater than a preset confidence threshold as a detected product, and output product information of the detected product.

[0029] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned shopping cart cluster environment information perception method when executing the computer program.

[0030] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned shopping cart cluster environment information perception method is implemented.

[0031] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned shopping cart cluster environment information perception method is implemented.

[0032] In an embodiment of the present invention, after receiving an information collection instruction, a video stream is obtained, and its own environmental information is collected at the same time; the information collection instruction is sent by the environment perception server to each shopping cart after obtaining the environmental information that needs to be collected for each shopping cart in the shopping cart cluster according to the accumulated shopping environment information and the position of each shopping cart, and the information collection instruction includes the environmental information that needs to be collected by the shopping cart; the information collection instruction and the video stream are input into the target detection model to obtain the commodity information of the commodity detected from the video stream; the own environmental information and the commodity information of the detected commodity are summarized to obtain the shopping environment information of the shopping cart; the target detection model includes: a selection mechanism decision module, which is used to determine the anchor box set and the corresponding feature vector corresponding to the input image corresponding to the video stream; a pre-trained model for image-text comparison, which is used to obtain at least one text feature and multiple image features according to the anchor box set and the corresponding feature vector and the information collection instruction; the confidence is calculated according to the at least one text feature and the multiple image features; the commodity with a confidence greater than a preset confidence threshold is determined as the detected commodity, and the commodity information of the detected commodity is output. Compared with the prior art, the present invention can send information collection instructions to each shopping cart to obtain surrounding environment information, so that each shopping cart can apply the target detection model to perform commodity detection oriented to the development target, that is, it can detect commodities of non-predefined types. This is because the target detection model screens the anchor point box through the selection mechanism decision module, and can perform image feature supervision and text feature supervision at the same time through the pre-training model for image-text comparison, which can understand rich visual concepts and natural language descriptions, and ultimately greatly improve the accuracy and efficiency of commodity detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0034] Figure 1 Flow chart of a method for sensing shopping cart cluster environment information according to an embodiment of the present invention;

[0035] Figure 2 This is an application flow chart of the environmental information library in an embodiment of the present invention;

[0036] Figure 3 It is a schematic diagram of the structure of the target detection model in an embodiment of the present invention;

[0037] Figure 4 Schematic diagram of a shopping cart cluster environment information sensing device according to an embodiment of the present invention;

[0038] Figure 5 Schematic diagram of a shopping cart cluster environment information perception system according to an embodiment of the present invention;

[0039] Figure 6 Schematic diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0041] The embodiment of the present invention is designed to deploy an open target detection model ArcDet for environmental perception on the shopping cart device side and a large-scale language model on the cloud. The large cloud model is used to control the perception mode of each small model in the shopping cart cluster and summarize the perception results to update the environmental information library to respond to questions from users. The ArcDet model is an open category-oriented target detection model modified on the basis of RT-DETR. It can detect the corresponding category based on the input text and is no longer limited to predefined categories. The environmental information obtained by the shopping carts in the form of clusters is cross-validated by nearby shopping carts, thereby improving the reliability of the detection results. The overall reliability, intelligence, and accuracy of the system are greatly improved.

[0042] Figure 1 This is a flow chart of a method for sensing environmental information of a shopping cart cluster in an embodiment of the present invention, which is applied to a shopping cart and includes:

[0043] Step 101, after receiving the information collection instruction, obtain the video stream and collect the environment information of the shopping cart itself; the information collection instruction is sent by the environment perception server to each shopping cart after obtaining the environment information that needs to be collected for each shopping cart in the shopping cart cluster according to the accumulated shopping environment information and the position of each shopping cart, and the information collection instruction includes the environment information that needs to be collected by the shopping cart;

[0044] Step 102, inputting the information collection instruction and the video stream into the target detection model to obtain the commodity information of the commodity detected from the video stream;

[0045] Step 103, summarizing the environment information of the shopping cart and the detected commodity information to obtain the shopping environment information of the shopping cart;

[0046] The target detection model includes:

[0047] A selection mechanism decision module is used to determine a set of anchor boxes and corresponding feature vectors corresponding to an input image corresponding to a video stream;

[0048] A pre-trained model for image-text comparison is used to obtain at least one text feature and multiple image features based on a set of anchor boxes and corresponding feature vectors and information collection instructions; calculate confidence based on the at least one text feature and multiple image features; determine a product with a confidence greater than a preset confidence threshold as a detected product, and output product information of the detected product.

[0049] Compared with the prior art, the present invention can send information collection instructions to each shopping cart to obtain surrounding environment information, so that each shopping cart can apply the target detection model to perform commodity detection oriented to the development target, that is, it can detect commodities of non-predefined types. This is because the target detection model screens the anchor point box through the selection mechanism decision module, and can perform image feature supervision and text feature supervision at the same time through the pre-training model for image-text comparison, which can understand rich visual concepts and natural language descriptions, and ultimately greatly improve the accuracy and efficiency of commodity detection.

[0050] In step 101, after receiving the information collection instruction, the video stream is obtained and the environment information of the shopping cart is collected at the same time; the information collection instruction is sent by the environment perception server to each shopping cart after the environment information that needs to be collected for each shopping cart in the shopping cart cluster is obtained according to the accumulated shopping environment information and the position of each shopping cart, and the information collection instruction includes the environment information that needs to be collected by the shopping cart;

[0051] Specifically, the shopping cart obtains the video stream through the deployed camera.

[0052] In one embodiment, the shopping environment information includes time, shopping cart location, product information, and shelf information; it is expressed as [K i ], where [K i ] represents the i-th environmental information, including time, shopping cart location, product information, shelf information, etc.

[0053] Product information includes the location, category, and confidence level to be explored;

[0054] Its own environmental information includes time, shopping cart location and shelf information; it is expressed as [C j ], where [C j ] represents the environment information of the j-th shopping cart;

[0055] The environmental information that the shopping cart needs to collect includes the list of products to be detected and the locations that need to be explored. The locations that need to be explored are the locations of the products in the supermarket.

[0056] The information collection instruction of the jth shopping cart can be expressed as prompt j .

[0057] When the shopping cart obtains the video stream, it also collects its own environmental information C j After the product information of the detected products is summarized, K i The information representation is uploaded to the environment information repository.

[0058] In one embodiment, the method further comprises:

[0059] The shopping environment information of the shopping cart is stored as the accumulated shopping environment information in the environment information library;

[0060] Based on the accumulated shopping environment information and the location of each shopping cart, the environment information that needs to be collected for each shopping cart in the shopping cart cluster is obtained, including:

[0061] Reading the accumulated shopping environment information from the environment information database;

[0062] The accumulated shopping environment information and the location of each shopping cart are input into the large language model to obtain the environmental information that needs to be collected for each shopping cart in the shopping cart cluster.

[0063] Figure 2 This is an application flow chart of the environmental information library in an embodiment of the present invention. In one embodiment, the method further includes:

[0064] Step 201, receiving an environmental information query input by a user;

[0065] Step 202, the accumulated shopping environment information and the environment information query input by the user are input into the large language model to obtain the shopping environment information corresponding to the environment information query.

[0066] It can be seen that users request relevant information in the shopping environment from the big model through the shopping cart, including product information acquisition and product recommendations. Product information includes product specifications, prices, locations, etc. According to the questions raised by the user, the big model integrates the existing environment information library, unifies the user questions and the environment information library into input text, and the big model provides answers and recommendations in a generated form. The core part is to obtain the big model input prompt based on the user questions and the structured environment information library. This part will be introduced in detail with examples.

[0067] For example, the following information items are stored in the environment information library:

[0068] July 14, clothing section of XX supermarket, short-sleeved shirt, 120 yuan, red;

[0069] July 18, clothing section of XX supermarket, Polo shirt, 240 yuan, dark blue;

[0070] July 18, XX supermarket clothing area, men’s trousers, 150 yuan, khaki;

[0071] July 19, XX supermarket fresh food area, crabs, 50 yuan / jin;

[0072] On June 15, in the fresh food section of XX supermarket, fresh shrimps were 20 yuan / catties;

[0073] July 18, YY Supermarket clothing section, men’s T-shirt, 60 yuan;

[0074] …

[0075] When a user at XX supermarket asks the following question on the smart device in the shopping cart: "I want to find a summer top suitable for a 175cm tall man", the information items within a certain time window in the past (for example, within half a month before the user asked the question) are first organized into a unified text according to time, and the user's question is added as the text input of the large model: XX supermarket's clothing area has red short-sleeved shirts at 120 yuan per piece, dark blue polo shirts at 240 yuan per piece, and men's trousers at 150 yuan per pair; XX supermarket's fresh food area has crabs at 50 yuan per piece and many other products. Please recommend a summer top for a man who is 175 cm tall, and explain the reason for the recommendation and the location of the product.

[0076] Note that in the above structured text, the fresh shrimp items from XX supermarket and the men's T-shirts from YY supermarket are excluded. The former is because the time of the product item exceeds the time window and there is a possibility that the product will be removed from the shelves, and the latter is because it does not match the XX supermarket where the user is located. The feedback of the large model to the above structured text input is as follows:

[0077] Recommended products:

[0078] 1. Dark Blue Polo Shirt:

[0079] Price: 240 yuan / piece

[0080] Recommended reason: Dark blue Polo shirts are classic and versatile, suitable for all occasions, from casual to business. The shape of the Polo shirt can well set off the figure, suitable for men of 175 cm. Dark blue is steady and durable, suitable for refreshing summer wear.

[0081] Location: Polo shirt counter in the clothing area

[0082] Red short sleeves:

[0083] Price: 120 yuan / piece

[0084] Recommended reason: The red short-sleeved shirt is bright in color, which can add vitality to the summer and is suitable for daily casual wear. The price is relatively affordable, suitable for consumers who pursue cost-effectiveness.

[0085] Location: Short-sleeve counter in clothing area

[0086] Choose the right top based on your personal preference and the occasion you need it for. If you need more variety, you can consider buying both and use them for different occasions.

[0087] On the display end of the shopping cart, in addition to feeding back the above text content to the user, the corresponding pictures will also be parsed, and the product image information collected in the aforementioned links will be displayed and fed back to the user, so as to facilitate the user's further selection and positioning.

[0088] In step 102, the information collection instruction and the video stream are input into the target detection model to obtain the commodity information of the commodity detected from the video stream;

[0089] The object detection model is an open-category object detection model that can detect related products in an image based on the input text.

[0090] Traditional object detection and segmentation models are often trained on predefined fixed categories of data. Object detectors trained on these data sets can only detect a limited number of objects and are powerless against objects that are not in the predefined categories. If you need to increase the ability to detect a new object, you can only incrementally learn and train the model by adding labeled data. However, the number of categories in real application scenarios far exceeds the data coverage and can be considered infinite. It is impossible to cover all categories through data annotation.

[0091] Unlike traditional target detectors (such as Faster RCNN, YOLO, etc.), the target detection model proposed in the embodiment of the present invention is not limited to predefined commodity categories and supports any commodity type described in text. Traditional target detectors can only detect 80 types of targets (such as bicycles, cars, etc.), and lose the ability to detect other types that are not included in these 80 categories, such as red jackets and milk powder. When detecting, the target detection model proposed in the embodiment of the present invention only needs to input the text description of the commodity to be detected.

[0092] The target detection model proposed in the embodiment of the present invention has the ability to detect open targets. During the model training process, the image encoder is used to supervise the output of the newly added feature branch so that it can extract the spatial image features in the corresponding area. The pre-trained model for image-text comparison aims to understand rich visual concepts and natural language descriptions through comparative learning of large-scale image-text pairs. A sub-center cosine loss function ArcLoss with a penalty term is proposed in the target detection model, which accelerates the convergence of training while enhancing the ability to distinguish between samples within and between classes. Compared with the cosine loss function and the cross entropy loss function, ArcLoss effectively reduces the burden of spatial image feature training for the pre-trained model for image-text comparison, making it possible. In order to improve the object detection ability of the target detection model, the embodiment of the present invention proposes a selection mechanism decision module so that the model has the ability to locate general targets.

[0093] Figure 3 is a schematic diagram of the structure of the target detection model in an embodiment of the present invention, Figure 3 In the training of the target detection model, a large image segmentation model, a fusion feature extraction module, a selection mechanism decision module and a pre-trained model for image-text comparison are required, while in the reasoning application, only a fusion feature extraction module, a selection mechanism decision module and a pre-trained model for image-text comparison are required.

[0094] In one embodiment, the fusion feature extraction module includes a backbone network and a feature enhancement network;

[0095] A backbone network, used to extract a first feature in an input image corresponding to a video stream;

[0096] The feature enhancement network is used to enhance the first feature to obtain a fusion feature.

[0097] In one embodiment, the selection mechanism decision module includes:

[0098] The encoding module is used to encode the fusion features and obtain multiple candidate anchor boxes {B j};

[0099] The decision mechanism module is used to calculate each candidate anchor box {B j} and each object mask {M i}, and filter out the anchor box set according to the overlap degree;

[0100] The decoding module is used to decode the anchor box set to obtain the anchor box set and the corresponding feature vector.

[0101] The overlap can be expressed as:

[0102] IoU ij = IoU ij(M i ,B j ) (1)

[0103] Among them, IoU ij is the candidate anchor box {B j} and object mask {M i}.

[0104] In order to ensure that the candidate box covers a larger range of objects, the anchor box is filtered as follows.

[0105] In one embodiment, the following formula is used to filter out the anchor point frame set according to the overlap degree:

[0106] (2)

[0107] Among them, B + is the filtered anchor box set, is the object mask M i Candidate anchor box with maximum overlap; IoU i* is the object mask M i The overlap between any candidate anchor box; IoU ik is the preset overlap threshold.

[0108] In order to ensure B + There are sufficient candidate boxes, and the image segmentation model under different prompting modes is used to extract the object mask in the input image. The image segmentation model has multiple prompting modes, namely point, box (SamPredictor), mask, text, and automatic full image segmentation. In the embodiment of the present invention, three prompting modes of point, box, and full image segmentation are used, and the object mask is extracted in the priority order of full image segmentation-point-box, and sorted according to the score, and the highest K M are taken. i As a screening target.

[0109] In one embodiment, the decision mechanism module is further used to:

[0110] Obtaining object masks output by a large image segmentation model in different prompting modes; the prompting modes include a point mode, a box mode, and a full image segmentation mode;

[0111] Extract object masks in the order of full image segmentation mode, point mode, and frame mode;

[0112] In the order of the extracted object masks, the overlap between each candidate anchor box and the object mask is calculated.

[0113] In one embodiment, the pre-trained model for image-text comparison includes:

[0114] An image encoder is used to intercept the input image I according to the anchor box set (box) to obtain multiple image features;

[0115] A text encoder, used to extract at least one text feature of the product list in the information collection instruction;

[0116] The analysis module is used to calculate the confidence level according to at least one text feature and multiple image features; determine the commodity with a confidence level greater than a preset confidence threshold as the detected commodity, and output the commodity information of the detected commodity.

[0117] Among them, the feature vector emd of the anchor box set can be expressed as:

[0118] Box,emd=Sel(q,M) (3)

[0119] Among them, M is the object mask, q is the fusion feature; Sel is the selection mechanism decision module, the feature vector corresponding to the emd anchor box set is the category branch; box is the anchor box set.

[0120] emd text =CP text (categories) (4)

[0121] emd image =CP image (I,box) (5)

[0122] Among them, emd text is the text feature, emd image is the image feature, and categories is the information collection instruction.

[0123] The target detection model proposed in the embodiment of the present invention adds a feature branch (feature vectors corresponding to the anchor box set) on the basis of the positioning branch (anchor box) to extract image features corresponding to the anchor box. During training, the output of the feature branch is supervised by the pre-extracted image features; during prediction, the pre-extracted text features of the category to be detected are used to determine its confidence in each category.

[0124] Text Encoder CP text Extract the text feature emd corresponding to the category name (information collection instruction) text , image encoder CP image Cut the box area in the original image I of the input image to obtain its image features emd image The decoder output emd is supervised for training, and the category text feature emd is also added textThe image features extracted here correspond to the original image range, which is expanded by 1.2 times while keeping the center of the target frame unchanged. This is because many objects are often difficult to identify when they lack background information.

[0125] In one embodiment, the target detection model further includes an image segmentation model, which is used during training but is not required during prediction applications. The image segmentation model is used to segment the input target image during training to obtain multiple object masks corresponding to each target image, represented as {M i}.

[0126] In the embodiment of the present invention, the training steps of the target detection model are as follows:

[0127] Acquire multiple target images;

[0128] Input multiple target images into the fusion feature extraction module to obtain the fusion feature corresponding to each target image;

[0129] Repeat the following steps until the total loss function value converges:

[0130] Each target image is input into the image segmentation model to obtain multiple object masks {M i};

[0131] The multiple object masks {M i} and the fusion feature input selection mechanism decision module to obtain the anchor box set and the corresponding feature vector emd;

[0132] The anchor box set and the corresponding feature vectors and information collection instructions are input into the pre-trained model for image-text comparison to obtain the product information of the detected products.

[0133] During training, the total loss function is used to determine the termination conditions of training. After reaching the preset convergence conditions, the total loss function value is determined to converge, the training is terminated, and a trained target detection model is obtained.

[0134] In one embodiment, the total loss function value is expressed by the following formula:

[0135] Loss=L box +L logit (emd,emd image )+ L logit (emd,emd text ) (6)

[0136] Among them, Loss is the total loss function value, L box is the loss function of the anchor box, L logitis the loss function for supervising the feature vector output, emd is the feature vector, emd image is the image feature, emd text is a text vector. box Contains GIoU and L1 loss functions. logit is the loss function for supervising the output of the feature branch. logit Give a description.

[0137] The loss function is modified by referring to the sub-center cosine distance (Sub-center cosine loss), which aims to better distinguish positive and negative samples in the spherical feature space. In the supervised training of the target detection model, the same category of targets may appear in the same training batch or the same image, and most of the labeled data comes from the image segmentation model extraction, and the category is unknown. Therefore, emd is used here. j Expand from a 1×512 vector to the closest 1×512×k vectors in the emd matrix, so , and make y i =i.

[0138] In one embodiment, the loss function for supervising the feature vector output is expressed by the following formula:

[0139]

[0140]

[0141] θ ii =arccos(max_pooling(emd i •emd i )

[0142] θ ij =arccos(max_pooling(emd i •emd image,j )

[0143] θ ik =arccos(max_pooling(emd i •emd text,k )

[0144] Among them, max_pooling is the pooling operation, arccos is the inverse cosine function, is the interval parameter between positive and negative samples, emd i is the i-th eigenvector, emd image,j is the jth image feature, emd text,k is the kth text vector.

[0145] in, , , k is the number of sub-centers. After max_pooling, an N×N matrix is ​​obtained. Its physical meaning can be understood as the minimum angle between the mask category features of each object and the feature clusters formed by k sub-centers. m is the penalty term for the distance between positive and negative samples, which aims to improve the ability to distinguish between inter-class / intra-class samples.

[0146] The embodiment of the present invention also proposes the idea of ​​cluster mutual assistance.

[0147] The position of the shopping cart is controlled by the user. Therefore, when the product detection instruction (including the product to be detected and the shelf position) is only sent to the shopping cart closest to the shelf position, there is a high probability that the instruction cannot be completed. For example, although the shopping cart is very close to the target location, it is pushed to other locations by the user; or the angle of the shopping cart causes it to be unable to align with the target shelf. Considering that there are multiple shopping carts in the same shopping environment, the same product detection instruction will be sent to 5 shopping carts for simultaneous execution and a timeout period T will be set. When any two of the 5 shopping carts detect the target product within the first preset time T after receiving the instruction, the instruction on all shopping carts is considered to be executed; if after the first preset time T, no or only one shopping cart detects the product information, all shopping carts discard the instruction and no longer execute it.

[0148] The purpose of selecting more than two shopping carts to complete the detection is to improve the accuracy of detection and recognition through cross-validation. The logic is: when shopping cart C detects the information of product A at the target position P through the camera, the information will be uploaded to the environmental information library, but will not be updated and stored immediately. Until another shopping cart other than C also detects the information of product A within the range of △p near position P, the information will be updated to the environmental information library. This logic is executed in the cloud, and the shopping cart is only responsible for executing the instructions in the instruction queue and uploading the results.

[0149] In one embodiment, the method further comprises:

[0150] If the shopping environment information of the shopping cart is obtained within a first preset time period from receiving the information collection instruction, the shopping environment information is fed back to the environment perception server, so that the environment perception server determines that the information collection is successful after receiving the shopping environment information fed back by at least two shopping carts, otherwise it is determined that the information collection has failed.

[0151] Specifically for a shopping cart, when it receives an instruction, it stores the instruction in its own instruction queue. In the working state where it is pushed by the user, it obtains its own position at a fixed time interval t, and then traverses all instructions to determine whether it has reached the position in the instruction. If it has reached its position, the product description in the instruction is added to the description of the product to be detected, and the instruction is removed from the instruction queue; if it has not reached its position, it is determined whether it has timed out, and the timed out instruction is removed from the queue, and the instruction that has not timed out is ignored. Finally, the products that have reached their positions in the instruction queue are summarized into a product description text. The current video stream is detected through the description text and the target detection model, and the detection results are uploaded to the environmental information library.

[0152] In one embodiment, the method further comprises:

[0153] storing the received information collection instructions in an instruction queue;

[0154] During the movement of the shopping cart, the location of the shopping cart is obtained every second preset time period;

[0155] Traverse the instruction queue to determine whether the shopping cart position meets the position that needs to be explored in the information collection instruction;

[0156] If it matches, add the commodity list in the matching information collection instruction to the description of the commodity to be detected, and remove the matching information collection instruction from the instruction queue; input the description of the commodity to be detected and the video stream into the target detection model to obtain the commodity information of the commodity detected from the video stream;

[0157] If not, determining whether the existence time of each information collection instruction exceeds a first preset time length;

[0158] If it exceeds, the excess information collection instructions will be removed from the message queue.

[0159] The embodiment of the present invention further proposes a shopping cart cluster environment information sensing device, the principle of which is similar to the shopping cart cluster environment information sensing method, which will not be repeated here.

[0160] Figure 4 This is a schematic diagram of a shopping cart cluster environment information sensing device according to an embodiment of the present invention, which is applied to a shopping cart and includes:

[0161] The information collection module 401 is used to obtain the video stream and collect its own environment information after receiving the information collection instruction. The information collection instruction is sent by the environment perception server to each shopping cart after the environment perception server obtains the environment information that needs to be collected for each shopping cart in the shopping cart cluster according to the accumulated shopping environment information and the position of each shopping cart. The information collection instruction includes the environment information that needs to be collected by the shopping cart;

[0162] The commodity acquisition module 402 is used to input the information collection instruction and the video stream into the target detection model to obtain commodity information of the commodity detected from the video stream;

[0163] The shopping environment information acquisition module 403 is used to aggregate its own environment information and the commodity information of the detected commodities to obtain the shopping environment information of the shopping cart;

[0164] The target detection model includes:

[0165] A selection mechanism decision module is used to determine a set of anchor boxes and corresponding feature vectors corresponding to an input image corresponding to a video stream;

[0166] A pre-trained model for image-text comparison is used to obtain at least one text feature and multiple image features based on a set of anchor boxes and corresponding feature vectors and information collection instructions; calculate confidence based on the at least one text feature and multiple image features; determine a product with a confidence greater than a preset confidence threshold as a detected product, and output product information of the detected product.

[0167] In one embodiment, the device further comprises a storage module, configured to:

[0168] The obtained shopping environment information of the shopping cart is stored in the environment information library as accumulated shopping environment information.

[0169] In one embodiment, the device further comprises a query module, configured to:

[0170] Receive environmental information query input by the user;

[0171] The accumulated shopping environment information and the environment information query input by the user are input into the large language model to obtain the shopping environment information corresponding to the environment information query.

[0172] In one embodiment, the device further comprises a feedback module, configured to:

[0173] If the shopping environment information of the shopping cart is obtained within a first preset time period from receiving the information collection instruction, the shopping environment information is fed back to the environment perception server, so that the environment perception server determines that the information collection is successful after receiving the shopping environment information fed back by at least two shopping carts, otherwise it is determined that the information collection has failed.

[0174] In one embodiment, the feedback module is further configured to:

[0175] storing the received information collection instructions in an instruction queue;

[0176] During the movement of the shopping cart, the location of the shopping cart is obtained every second preset time period;

[0177] Traverse the instruction queue to determine whether the shopping cart position meets the position that needs to be explored in the information collection instruction;

[0178] If it matches, add the commodity list in the matching information collection instruction to the description of the commodity to be detected, and remove the matching information collection instruction from the instruction queue; input the description of the commodity to be detected and the video stream into the target detection model to obtain the commodity information of the commodity detected from the video stream;

[0179] If not, determining whether the existence time of each information collection instruction exceeds a first preset time length;

[0180] If it exceeds, the excess information collection instructions will be removed from the message queue.

[0181] The embodiment of the present invention also provides a shopping cart cluster environment information perception system. Figure 5 A schematic diagram of a shopping cart cluster environment information perception system according to an embodiment of the present invention, comprising: an environment perception server 501 and a shopping cart 502;

[0182] Context-aware servers for:

[0183] According to the accumulated shopping environment information and the location of each shopping cart, the environment information that needs to be collected for each shopping cart in the shopping cart cluster is obtained;

[0184] Sending an information collection instruction to each shopping cart, wherein the information collection instruction includes environmental information that the shopping cart needs to collect;

[0185] The shopping cart is used for:

[0186] After receiving the information collection instruction, it obtains the video stream and collects its own environmental information at the same time;

[0187] Input the information collection instruction and the video stream into the target detection model to obtain the commodity information of the commodity detected from the video stream;

[0188] Aggregate the own environment information and the detected commodity information to obtain the shopping environment information of the shopping cart;

[0189] The target detection model includes:

[0190] A selection mechanism decision module is used to determine a set of anchor boxes and corresponding feature vectors corresponding to an input image corresponding to a video stream;

[0191] A pre-trained model for image-text comparison is used to obtain at least one text feature and multiple image features based on a set of anchor boxes and corresponding feature vectors and information collection instructions; calculate confidence based on the at least one text feature and multiple image features; determine a product with a confidence greater than a preset confidence threshold as a detected product, and output product information of the detected product.

[0192] In summary, the method, device, and system proposed in the embodiments of the present invention have the following beneficial effects:

[0193] 1. Use large language models and target detection models to design a complete environmental information perception process;

[0194] 2. The object detection model deployed on the shopping cart belongs to the open object detection category. Different from the common object detection methods, its advantage is that it can detect objects described in any text, not limited to predefined categories;

[0195] 3. Cluster mutual assistance: The concept of shopping cart cluster is proposed for the first time. Through collaboration and cross-validation, the accuracy of obtaining perception information is improved to complete the extraction of environmental information.

[0196] An embodiment of the present invention further provides a computer device, Figure 6 It is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 600 includes a memory 610, a processor 620, and a computer program 630 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 630, the above-mentioned shopping cart cluster environment information perception method is implemented.

[0197] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned shopping cart cluster environment information perception method is implemented.

[0198] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned shopping cart cluster environment information perception method is implemented.

[0199] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0200] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0203] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for perceiving environmental information of a shopping cart cluster, characterized in that: Applied to shopping cart, including: After receiving the information collection instruction, the video stream is obtained and the environment information of the shopping cart is collected at the same time; the information collection instruction is sent by the environment perception server to each shopping cart after the environment information that needs to be collected for each shopping cart in the shopping cart cluster is obtained based on the accumulated shopping environment information and the position of each shopping cart, and the information collection instruction includes the environment information that needs to be collected by the shopping cart; Input the information collection instruction and the video stream into the target detection model to obtain the commodity information of the commodity detected from the video stream; Aggregate the own environment information and the detected commodity information to obtain the shopping environment information of the shopping cart; The target detection model includes: A selection mechanism decision module is used to determine a set of anchor boxes and corresponding feature vectors corresponding to an input image corresponding to a video stream; A pre-trained model for image-text comparison, used to obtain at least one text feature and multiple image features according to an anchor box set and corresponding feature vectors and information collection instructions; calculate confidence according to the at least one text feature and multiple image features; determine a product with a confidence greater than a preset confidence threshold as a detected product, and output product information of the detected product; The selection mechanism decision module includes: The encoding module is used to encode the fused features to obtain multiple candidate anchor boxes; The decision mechanism module is used to calculate the overlap between each candidate anchor box and each object mask, and filter out the anchor box set based on the overlap; The decoding module is used to decode the anchor box set to obtain the anchor box set and the corresponding feature vector.

2. The method according to claim 1, characterized in that Shopping environment information includes time, shopping cart location, product information, and shelf information; Product information includes the location, category, and confidence level to be explored; Its own environmental information includes time, shopping cart location and shelf information; The environmental information that the shopping cart needs to collect includes the list of products to be detected and the locations that need to be explored.

3. The method according to claim 1, characterized in that Also includes: The shopping environment information of the shopping cart is stored as the accumulated shopping environment information in the environment information library; Based on the accumulated shopping environment information and the location of each shopping cart, the environment information that needs to be collected for each shopping cart in the shopping cart cluster is obtained, including: Reading the accumulated shopping environment information from the environment information database; The accumulated shopping environment information and the location of each shopping cart are input into the large language model to obtain the environmental information that needs to be collected for each shopping cart in the shopping cart cluster.

4. The method according to claim 2, characterized in that Also includes: Receive environmental information query input by the user; The accumulated shopping environment information and the environment information query input by the user are input into the large language model to obtain the shopping environment information corresponding to the environment information query.

5. The method according to claim 1, characterized in that The object detection model also includes a fusion feature extraction module; The fusion feature extraction module includes a backbone network and a feature enhancement network; A backbone network, used to extract a first feature in an input image corresponding to a video stream; The feature enhancement network is used to enhance the first feature to obtain a fusion feature.

6. The method according to claim 1, characterized in that The decision-making mechanism module is also used to: Obtaining object masks output by a large image segmentation model in different prompting modes; the prompting modes include a point mode, a box mode, and a full image segmentation mode; Extract object masks in the order of full image segmentation mode, point mode, and frame mode; In the order of the extracted object masks, the overlap between each candidate anchor box and the object mask is calculated.

7. The method according to claim 1, characterized in that The following formula is used to filter out the anchor box set according to the overlap: in, is the filtered anchor box set, For the object mask Candidate anchor box with maximum overlap; Mask the object The degree of overlap with any candidate anchor box; is the preset overlap threshold.

8. The method according to claim 2, characterized in that Pre-trained models for image-text comparison include: An image encoder is used to intercept the input image according to the anchor box set to obtain multiple image features; A text encoder, used to extract at least one text feature of the product list in the information collection instruction; The analysis module is used to calculate the confidence level according to at least one text feature and multiple image features; determine the commodity with a confidence level greater than a preset confidence threshold as the detected commodity, and output the commodity information of the detected commodity.

9. The method according to claim 1, characterized in that The object detection model also includes a large image segmentation model; The training steps of the target detection model are as follows: Acquire multiple target images; Input multiple target images into the fusion feature extraction module to obtain the fusion feature corresponding to each target image; Repeat the following steps until the total loss function value converges: Input each target image into the image segmentation model to obtain multiple object masks corresponding to each target image; Input multiple object masks and fusion features corresponding to each target image into the selection mechanism decision module to obtain an anchor box set and a corresponding feature vector; The anchor box set and the corresponding feature vectors and information collection instructions are input into the pre-trained model for image-text comparison to obtain the product information of the detected products.

10. The method according to claim 9, characterized in that The total loss function value is expressed as follows: in, is the total loss function value, is the loss function of the anchor box, is the loss function for supervising the feature vector output, is the feature vector, is the image feature, is a text vector.

11. The method according to claim 10, characterized in that The loss function for supervising the feature vector output is expressed as follows: Among them, max_pooling is the pooling operation, is the arccosine function, is the interval parameter between positive and negative samples, is the i-th eigenvector, is the jth image feature, is the kth text vector.

12. The method according to claim 1, characterized in that Also includes: If the shopping environment information of the shopping cart is obtained within a first preset time period from receiving the information collection instruction, the shopping environment information is fed back to the environment perception server, so that the environment perception server determines that the information collection is successful after receiving the shopping environment information fed back by at least two shopping carts, otherwise it is determined that the information collection has failed.

13. The method according to claim 12, characterized in that Also includes: storing the received information collection instructions in an instruction queue; During the movement of the shopping cart, the location of the shopping cart is obtained every second preset time period; Traverse the instruction queue to determine whether the shopping cart position meets the position that needs to be explored in the information collection instruction; If it matches, add the commodity list in the matching information collection instruction to the description of the commodity to be detected, and remove the matching information collection instruction from the instruction queue; input the description of the commodity to be detected and the video stream into the target detection model to obtain the commodity information of the commodity detected from the video stream; If not, determining whether the existence time of each information collection instruction exceeds a first preset time length; If it exceeds, the excess information collection instructions will be removed from the message queue.

14. A shopping cart cluster environment information sensing device, characterized in that: Applied to shopping cart, including: An information collection module, which is used to obtain the video stream and collect its own environment information after receiving an information collection instruction. The information collection instruction is sent by the environment perception server to each shopping cart after the environment perception server obtains the environment information that needs to be collected for each shopping cart in the shopping cart cluster based on the accumulated shopping environment information and the position of each shopping cart. The information collection instruction includes the environment information that needs to be collected by the shopping cart; A commodity acquisition module, used to input the information collection instruction and the video stream into the target detection model to obtain commodity information of the commodity detected from the video stream; A shopping environment information acquisition module is used to aggregate its own environment information and the commodity information of the detected commodities to obtain the shopping environment information of the shopping cart; The target detection model includes: A selection mechanism decision module is used to determine a set of anchor boxes and corresponding feature vectors corresponding to an input image corresponding to a video stream; A pre-trained model for image-text comparison, used to obtain at least one text feature and multiple image features according to an anchor box set and corresponding feature vectors and information collection instructions; calculate confidence according to the at least one text feature and multiple image features; determine a product with a confidence greater than a preset confidence threshold as a detected product, and output product information of the detected product; The selection mechanism decision module includes: The encoding module is used to encode the fused features to obtain multiple candidate anchor boxes; The decision mechanism module is used to calculate the overlap between each candidate anchor box and each object mask, and filter out the anchor box set based on the overlap; The decoding module is used to decode the anchor box set to obtain the anchor box set and the corresponding feature vector.

15. A shopping cart cluster environment information perception system, characterized in that: include: context-aware servers and shopping carts; Context-aware servers for: According to the accumulated shopping environment information and the location of each shopping cart, the environment information that needs to be collected for each shopping cart in the shopping cart cluster is obtained; Sending an information collection instruction to each shopping cart, wherein the information collection instruction includes environmental information that the shopping cart needs to collect; The shopping cart is used for: After receiving the information collection instruction, it obtains the video stream and collects its own environmental information at the same time; Input the information collection instruction and the video stream into the target detection model to obtain the commodity information of the commodity detected from the video stream; Aggregate the own environment information and the detected commodity information to obtain the shopping environment information of the shopping cart; The target detection model includes: A selection mechanism decision module is used to determine a set of anchor boxes and corresponding feature vectors corresponding to an input image corresponding to a video stream; A pre-trained model for image-text comparison, used to obtain at least one text feature and multiple image features according to an anchor box set and corresponding feature vectors and information collection instructions; calculate confidence according to the at least one text feature and multiple image features; determine a product with a confidence greater than a preset confidence threshold as a detected product, and output product information of the detected product; The selection mechanism decision module includes: The encoding module is used to encode the fused features to obtain multiple candidate anchor boxes; The decision mechanism module is used to calculate the overlap between each candidate anchor box and each object mask, and filter out the anchor box set based on the overlap; The decoding module is used to decode the anchor box set to obtain the anchor box set and the corresponding feature vector.

16. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 11 is implemented.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

18. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images