Commodity information determination method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a commodity information determination method, device, electronic equipment and storage medium. The commodity information determination method comprises: determining a first commodity image corresponding to a commodity in a first image and a second commodity image corresponding to a commodity in a second image respectively by using an open set detection model, wherein the first image is an image of a target region at a first time, the second image is an image of the target region at a second time, and the first time is before the second time; comparing the first commodity image corresponding to the commodity in the first image and the second commodity image corresponding to the commodity in the second image to determine a third commodity image corresponding to a commodity out of stock reduced in the second image compared with the first image; and identifying the third commodity image by using a visual recognition model to determine commodity information of the commodity out of stock corresponding to the third commodity image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a commodity information determination method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, convenience stores and supermarkets and other commercial entities usually adopt a self-service mode for customers to select commodities, and the commodities are placed on shelves for customers to select and purchase.

[0003] In the process of commodity sales, the commodities on the shelves need to be checked and replenished in time, so as to reduce the shortage of commodities on the shelves.

[0004] Therefore, how to determine the commodity information on the shelves in time, such as the name and quantity of the out-of-stock commodities, is an urgent problem to be solved. SUMMARY

[0005] The present application provides a commodity information determination method and device, electronic equipment and storage medium.

[0006] According to a first aspect of the present application, a commodity information determination method is provided, which comprises:

[0007] The open set detection model is used to determine the first commodity image corresponding to the commodity in the first image and the second commodity image corresponding to the commodity in the second image, respectively, wherein the first image is an image of a target region at a first time, and the second image is an image of the target region at a second time, wherein the first time is before the second time;

[0008] The first commodity image corresponding to the commodity in the first image and the second commodity image corresponding to the commodity in the second image are compared to determine the third commodity image corresponding to the out-of-stock commodity that is reduced in the second image compared with the first image;

[0009] The visual recognition model is used to identify the third commodity image to determine the commodity information of the out-of-stock commodity corresponding to the third commodity image.

[0010] In some embodiments, the open set detection model is used to determine the first commodity image corresponding to the commodity in the first image and the second commodity image corresponding to the commodity in the second image, respectively, comprising:

[0011] The open set detection model is used to identify the commodity in the first image, and the first commodity image corresponding to the identified commodity in the first image is marked with a predetermined mark;

[0012] The open set detection model is used to identify the commodity in the second image, and the second commodity image corresponding to the identified commodity in the second image is marked with a predetermined mark;

[0013] The comparing the first product image corresponding to the product in the first image and the second product image corresponding to the product in the second image comprises:

[0014] The comparing the first product image marked with the predetermined identifier and the second product image marked with the predetermined identifier.

[0015] In some embodiments, the method further comprises:

[0016] Training the open set detection model by using a first training product image of a first non-standard appearance product of a predetermined product type.

[0017] In some embodiments, the training the open set detection model by using the first training product image of the first non-standard appearance product of the predetermined product type comprises at least one of:

[0018] Training the output of the open set detection model by taking the first training product image as the input of the open set detection model and taking the product type of the first non-standard appearance product as the annotation information of the first training product image.

[0019] Training the output of the open set detection model by taking the first training product image as the input of the open set detection model and specifying the position information of the product type of the first non-standard appearance product in the image description of the first training product image.

[0020] In some embodiments, the method further comprises:

[0021] In response to the open set detection model being able to identify the product type of a second non-standard appearance product in a product image, determining that the training of the open set detection model is completed.

[0022] In some embodiments, the method further comprises:

[0023] Training a visual recognition model capable of identifying a first classification of products to enable the visual recognition model to identify a second classification of products, wherein the second classification is a sub-classification of the first classification.

[0024] In some embodiments, the training the visual recognition model capable of identifying the first classification of products to enable the visual recognition model to identify the second classification of products comprises:

[0025] Training the visual recognition model by taking a second training product image as the input of the visual recognition model and taking a product description containing a second classification description of the product in the second training product image as the output of the visual recognition model.

[0026] In response to the visual recognition model identifying the second category of the commodity in the unlabeled commodity image, it is determined that the visual recognition model has completed training.

[0027] According to a second aspect of the embodiments of the present disclosure, a commodity information determination apparatus is provided, and the apparatus comprises a processing module, wherein the processing module is configured to:

[0028] The open set detection model is used to determine a first commodity image corresponding to the commodity in the first image and a second commodity image corresponding to the commodity in the second image, wherein the first image is an image of a target region at a first time point, and the second image is an image of the target region at a second time point, wherein the first time point is before the second time point;

[0029] The first commodity image corresponding to the commodity in the first image and the second commodity image corresponding to the commodity in the second image are compared, and a third commodity image corresponding to a commodity out of stock in the second image compared with the first image is determined;

[0030] The visual recognition model is used to identify the third commodity image, and commodity information of a commodity out of stock corresponding to the third commodity image is determined.

[0031] In some embodiments, the processing module is specifically configured to:

[0032] The open set detection model is used to identify the commodity in the first image, and a first commodity image corresponding to the identified commodity in the first image is marked with a predetermined identifier;

[0033] The open set detection model is used to identify the commodity in the second image, and a second commodity image corresponding to the identified commodity in the second image is marked with a predetermined identifier;

[0034] The first commodity image marked with the predetermined identifier and the second commodity image marked with the predetermined identifier are compared.

[0035] In some embodiments, the apparatus further comprises a training module,

[0036] The training module is configured to train the open set detection model using a first training commodity image of a first non-standard appearance commodity labeled as a predetermined commodity type.

[0037] In some embodiments, the training module is specifically configured to at least one of the following:

[0038] The first training commodity image is taken as an input of the open set detection model, a commodity type of the first non-standard appearance commodity is taken as labeled information of the first training commodity image, and an output of the open set detection model is trained;

[0039] The first training commodity image is input into the open set detection model, and position information of a commodity type of the first non-standard appearance commodity in an image description of the first training commodity image is specified, and the output of the open set detection model is trained.

[0040] In some embodiments, the training module is further configured to: in response to the open set detection model being able to identify a commodity type of a second non-standard appearance commodity in a commodity image, determine that the open set detection model is trained.

[0041] In some embodiments, the apparatus further includes a training module,

[0042] The training module is configured to train the visual recognition model capable of identifying a first classification of commodities, so that the visual recognition model is capable of identifying a second classification of commodities, wherein the second classification is a sub-classification of the first classification.

[0043] In some embodiments, the training module is specifically configured to:

[0044] The second training commodity image is input into the visual recognition model, a commodity description containing a second classification description of the commodity in the second training commodity image is input into the visual recognition model, and the visual recognition model is trained.

[0045] In response to the visual recognition model identifying the second classification of the commodity in the unlabeled commodity image, it is determined that the visual recognition model is trained.

[0046] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, and the electronic device includes:

[0047] One or more processors;

[0048] The processor is configured to invoke instructions to cause the electronic device to perform the commodity information determination method of the first aspect.

[0049] According to a fourth aspect of embodiments of the present disclosure, a storage medium is provided, and the storage medium stores instructions, when the instructions run on an electronic device, cause the electronic device to perform the commodity information determination method of the first aspect.

[0050] A commodity information determination method, apparatus, electronic device, and storage medium are provided according to embodiments of the present disclosure. The commodity information determination method includes: determining, by using an open set detection model, a first commodity image corresponding to a commodity in a first image and a second commodity image corresponding to a commodity in a second image, respectively, wherein the first image is an image of a target region at a first time, the second image is an image of the target region at a second time, and the first time is before the second time; comparing the first commodity image corresponding to the commodity in the first image and the second commodity image corresponding to the commodity in the second image to determine a third commodity image corresponding to a commodity out of stock in the second image compared with the first image; identifying the third commodity image by using a visual recognition model to determine commodity information of the commodity out of stock corresponding to the third commodity image. In this way, by using the open set detection model, known and unknown categories of commodities in the image can be determined without labeling and training all types of commodities. Further, comparison of commodities in target region images at different times can be implemented to determine commodities out of stock, and the visual recognition model can be used to determine commodity types and other information of the commodities out of stock. On the one hand, the time for labeling all types of commodities and training the recognition model can be shortened, thereby saving costs. The commodities do not need to be labeled and trained before being put on the market, thereby improving the real-time performance of putting commodities on the market. On the other hand, the open set detection model does not need to accurately determine the types of commodities, thereby improving the recognition speed of the open set detection model. The visual recognition model is used to identify the commodities out of stock, thereby improving the speed of determining the commodities out of stock. On the other hand, since the open set detection model only needs to distinguish between commodities and non-commodities, the identification error caused by the fine-grained commodity type recognition of the recognition model can be reduced, thereby improving the accuracy of determining the commodities out of stock. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a schematic diagram of an application environment of a commodity information determination method according to an embodiment;

[0052] Figure 2 is a flowchart of a commodity information determination method according to an embodiment;

[0053] Figure 3 is a schematic diagram of a target region first image according to an embodiment;

[0054] Figure 4 is a schematic diagram of a target region second image according to an embodiment;

[0055] Figure 5 is a schematic diagram of a first commodity image in a first image according to an embodiment;

[0056] Figure 6 is a schematic diagram of a second commodity image in a second image according to an embodiment;

[0057] Figure 7 FIG. 3 is a third schematic view of a product image, according to an embodiment;

[0058] Figure 8 FIG. 4 is a flowchart of a method for detecting out-of-stock of products on a shelf, according to an embodiment;

[0059] Figure 9 FIG. 5 is a schematic view of a running flow of a system for detecting out-of-stock of products on a shelf, according to an embodiment;

[0060] Figure 10 FIG. 6 is a schematic view of a structure of a device for determining product information, according to an embodiment;

[0061] Figure 11 FIG. 7 is a schematic view of a structure of an electronic device, according to an embodiment. DETAILED DESCRIPTION

[0062] In order to make the technical solutions and advantages of the present application more obvious and understandable, the following will be described in detail by way of specific embodiments. The drawings are not necessarily drawn to scale, and local features can be enlarged or reduced to more clearly show the details of the local features. Unless otherwise defined, the technical and scientific terms used herein have the same meaning as the technical and scientific terms in the technical field to which the present application belongs.

[0063] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing some steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation manners in an embodiment can be combined arbitrarily; in addition, the embodiments can be combined arbitrarily, for example, the steps of different embodiments or part of the steps of different embodiments can be combined arbitrarily, an embodiment can be combined with the optional implementation manners of other embodiments arbitrarily.

[0064] In the embodiments of the present disclosure, the terms and / or descriptions of the embodiments are consistent and can be referred to each other if there is no special description and logical conflict, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0065] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and not as a limitation on the present disclosure.

[0066] In the embodiments of the present disclosure, an element represented in a singular form, such as "a", "an", "the", "said", "the aforementioned", "the foregoing", "this", and the like, unless otherwise specified, can represent "one and only one", or can represent "one or more", "at least one", and the like. For example, in the case of using articles such as "a", "an", "the" in English in translation, the noun after the article can be understood as a singular expression, or can be understood as a plural expression.

[0067] In the embodiments of the present disclosure, "plurality" refers to two or more.

[0068] In some embodiments, the terms "at least one of", "one or more of", "a plurality of", "multiple", and the like can be replaced with each other.

[0069] In some embodiments, the description manner of "at least one of A, B", "A and / or B", "A in one case, B in another case", "one case A, another case B", and the like can include the following technical solutions according to the case: A in some embodiments (A is executed regardless of B); B in some embodiments (B is executed regardless of A); A and B are selectively executed in some embodiments (A and B are selected from A and B); A and B in some embodiments (A and B are executed). When there are more branches such as A, B, C, and the like, it is similar to the above.

[0070] In some embodiments, the description manner of "A or B" and the like can include the following technical solutions according to the case: A in some embodiments (A is executed regardless of B); B in some embodiments (B is executed regardless of A); A and B are selectively executed in some embodiments (A and B are selected from A and B). When there are more branches such as A, B, C, and the like, it is similar to the above.

[0071] The prefix words of "first", "second" and the like in the embodiments of the present disclosure are merely used to distinguish different description objects, and do not constitute limitation on the position, sequence, priority, value or content of the description objects. The description objects are described in the claims or embodiments in the context of the description, and should not be construed as redundant limitation because of the use of the prefix words. For example, the ordinal words in front of the description objects "field" in "first field" and "second field" do not limit the position or sequence between the "fields", and "first" and "second" do not limit whether the "fields" modified thereby are in the same message or not, nor limit the sequence of "first field" and "second field". For another example, the ordinal words in front of the description objects "level" in "first level" and "second level" do not limit the priority between the "levels". For another example, the value of the description objects is not limited by the ordinal words, and can be one or more. For example, "first device", in which the value of "device" can be one or more. In addition, the objects modified by different prefix words can be the same or different, for example, the description objects are "device", and "first device" and "second device" can be the same device or different devices, and the types thereof can be the same or different; for another example, the description objects are "information", and "first information" and "second information" can be the same information or different information, and the contents thereof can be the same or different.

[0072] In some embodiments, "including A", "containing A", "for indicating A", "carrying A" can be interpreted as directly carrying A, or indirectly indicating A.

[0073] In some embodiments, the terms of "…", "determining …", "in the case of …", "when …", "when …", "if …", "if …" and the like can be replaced with each other.

[0074] In some embodiments, the terms of "greater than", "greater than or equal to", "not less than", "more than", "more than or equal to", "not less than", "higher than", "higher than or equal to", "not lower than", "above" and the like can be replaced with each other, and the terms of "less than", "less than or equal to", "not greater than", "less than", "less than or equal to", "not more than", "lower than", "lower than or equal to", "not higher than", "below" and the like can be replaced with each other.

[0075] In some embodiments, the apparatus and the like can be interpreted as physical or virtual, and the name thereof is not limited to the name recorded in the embodiments. The terms of "apparatus", "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject" and the like can be replaced with each other.

[0076] In addition, each element, each row, or each column in the table of the embodiments of the present disclosure can be implemented as an independent embodiment, and any combination of any element, any row, or any column can also be implemented as an independent embodiment.

[0077] In some embodiments, the shelf commodity out-of-stock detection can be implemented by using a commodity identification model. The specific implementation method includes: (1) first, label the type of each commodity, and train a commodity identification model; (2) shoot a shelf image, and generate a shelf display comparison image; (3) use the commodity identification model to identify the commodity information (such as the commodity type) of the commodity in the current shelf and the commodity information (such as the commodity type) of the commodity in the display comparison image, and then compare and find out the out-of-stock commodity. It requires that the commodity identification model can directly identify the commodity information of the commodity in the shelf image.

[0078] The above-mentioned shelf commodity out-of-stock detection method using a commodity identification model mainly has the following problems:

[0079] 1. Difficult to adapt to the rapid new of commodities: the existing method mainly depends on the commodity identification model, and the accurate identification of the identification network needs the labeling of data samples. For some convenience stores and chain stores, the new of commodities is fast, and each new sample needs to make a training data set, and then perfect the commodity identification model, which increases the labor cost and affects the real-time performance.

[0080] 2. Low generalization of commodities: usually the same commodity will be subdivided into different sub-classes because of taste and packaging, and there may be packaging differences between different sub-classes, which will cause the existing commodity identification model to fail, and data needs to be supplemented in time. In actual situation, it is impossible to exhaustively train all commodities, so the failure rate of the commodity identification model is high.

[0081] 3. By training a specific commodity identification model one by one, when the commodity type increases or new commodities need to be recorded, several days or weeks of data are usually needed for commodity labeling or training.

[0082] And the open set detection model and the visual large model of the technical solution are used in cooperation with the implementation steps of the technical solution, when the commodity type increases or new commodities need to be recorded, the identification and detection period of the commodity is greatly shortened to minutes or hours, which greatly improves the speed of commodity new while ensuring the quality of commodity identification.

[0083] Figure 1 is a schematic diagram of an application environment of a commodity information determination method provided by the embodiments of the present disclosure, as Figure 1 shown, the application environment at least includes an information processing device 10 and an image acquisition device 20 (such as a terminal).

[0084] The information processing device 10 can be used as an environment of a bearing service, through which a service (such as a data processing service) running thereon processes image data transmitted by a user through the image collection device 20. The information processing device 10 can also send a processing result obtained by processing the image data to a terminal to present to the user. The information processing device 10 can be a standalone computing service device, a server cluster or a distributed system composed of multiple computing service devices, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN), and big data and artificial intelligence platform.

[0085] The image collection device 20 can include, but is not limited to, a smartphone, a tablet computer, a notebook computer, a digital assistant, a smart wearable device, a video camera, and a camera, and other electronic devices with image collection capabilities.

[0086] In addition, it should be noted that, Figure 1 The above is only an application environment of the commodity information determination method, and in actual application, other application environments can also be included, for example, more terminals, cameras, etc.

[0087] In the embodiments of the present disclosure, the above information processing devices 10 and / or the information processing device 10 and the image collection device 20 can be directly or indirectly connected through wired or wireless communication, and the present disclosure is not limited thereto.

[0088] The present disclosure provides a commodity information determination method, as shown in the figure, Figure 2 The commodity information determination method includes:

[0089] Step 201: using an open set detection model to determine a first commodity image corresponding to a commodity in a first image and a second commodity image corresponding to a commodity in a second image, respectively, wherein the first image is an image of a target region at a first time, and the second image is an image of the target region at a second time, wherein the first time is before the second time;

[0090] Step 202: comparing the first commodity image corresponding to the commodity in the first image and the second commodity image corresponding to the commodity in the second image, to determine a third commodity image corresponding to an out-of-stock commodity that is reduced in the second image compared with the first image;

[0091] Step 203: using a visual recognition model to recognize the third commodity image, to determine commodity information of the out-of-stock commodity corresponding to the third commodity image.

[0092] Here, the method for determining product information can be executed by the information processing device as shown in Figure 1

[0093] The first image and the second image can be images captured by an image capturing device. The image capturing device can send the captured images to the information processing device for determining the product information.

[0094] In one possible implementation, the image capturing device and the information processing device can be the same device.

[0095] In one possible implementation, the image capturing device and the information processing device can be different devices, and the images captured by the image capturing device can be transmitted to the information processing device through wired and / or wireless manner.

[0096] In one possible implementation, the target area can include a shelf area for placing products.

[0097] In one possible implementation, the first image can be a frame image in a video, and the second image can also be a frame image in the video. For example, the image capturing device can capture a video from the first time to the second time, and capture a frame image at the first time from the video as the first image, and capture a frame image at the second time from the video as the second image.

[0098] In one possible implementation, there can be multiple target areas, and different target areas can be distinguished by different identifiers. The product information of the out-of-stock product in the same target area can be determined by comparing the first image and the second image of the target area.

[0099] In one possible implementation, the out-of-stock product can be a product removed (e.g., taken away) from the target area between the first time and the second time.

[0100] For example, the first image can be an initial image of the target area, and the second image can be an image of the target area after a period of time.

[0101] As shown in Figure 3 , the first image can be an initial image of the target area at the first time. There are more products in the first image. As shown in Figure 4 , the second image can be an image of the target area at the second time. Some products in the target area are taken away in the second image. Figure 3 and Figure 4 The images of the target area in

[0102] ​The first image and the second image can be captured by the image capturing device at the same position and with the same angle. The first image and the second image can also be captured by the image capturing device at different positions and / or with different angles.

[0103] Here, the open-set detection model can be used to identify a first product image of the product from the first image, and a second product image of the product from the second image.

[0104] In one possible implementation, the open-set detection model performs coarse-grained recognition. The open-set detection model can distinguish the objects in the first image and the second image into products and non-products, so as to identify the products from the first image and the second image, and thus strip the products from the first image or the second image. For example, the open-set detection model can distinguish the products in the first image and the second image from non-products (such as shelves, people, sundries, etc.).

[0105] In one possible implementation, the open-set detection model can not identify the specific category of the product.

[0106] Here, the use of the open-set detection model to identify the product can solve the limitations of the traditional target detection method (closed-set target detection) in an open environment. Unlike the traditional closed-set target detection, the open-set detection model can identify targets of known categories, and can also identify and distinguish targets of unknown categories. For example, the open-set detection model can identify unknown products.

[0107] By using the open-set detection model to identify the products in the first image and the second image, the products on the shelves can be coarsely recognized, products outside the training samples can be identified, and products in non-standard states (such as different shapes and different packaging) can also be identified without the need for model training for new products. The training cost required by the traditional target detection method is effectively reduced.

[0108] For example, the same product is usually divided into different subcategories due to different tastes and packaging, and there may be packaging differences between different subcategories. The open-set detection model can identify products with different packaging from the image, so that the products can be stripped from the first image or the second image. Without the need for training for different packaging.

[0109] In one possible implementation, the first image can have one or more products, and therefore the open-set detection model can determine one or more first product images. The second image can have one or more products, and therefore the open-set detection model can determine one or more second product images.

[0110] In one possible implementation, one product corresponds to one product image.

[0111] As an example, Figure 5 The open set detection model is identified Figure 3 The first product image of the plurality of products obtained in the commodity is shown as a product shadow. As Figure 6 The open set detection model is identified Figure 4 The second product image of the plurality of products obtained in the commodity is shown as a product shadow.

[0112] After determining the first product image in the first image and the second product image in the second image, the first product image and the second product image can be compared to determine the third product image in the second image that is reduced compared to the first product image. The product corresponding to the third product image in the second image that is reduced compared to the first product image is the product taken away between the first time and the second time, that is, the out-of-stock product.

[0113] After determining the third product image, the product information of the corresponding out-of-stock product can be determined based on the third product image.

[0114] In one possible implementation, image recognition technology can be used to identify the out-of-stock product in the third product image.

[0115] The visual recognition model can include a machine learning model. The visual recognition model can be trained using product images and text feature descriptions, etc. to enable the visual recognition model to identify the product information of the out-of-stock product through the image feature information and the text information of the product in the third product image.

[0116] In some embodiments, the product information includes at least one of the following: the name of the out-of-stock product, the product type of the out-of-stock product.

[0117] In one possible implementation, the statistical information of the out-of-stock product can be determined based on the determined product information. For example, based on the product information determined from all third product images, the product name of the out-of-stock product, the out-of-stock quantity of the out-of-stock product, etc. can be determined.

[0118] Thus, by the open set detection model, the known and unknown classified goods in the image can be determined without labeling and training all kinds of goods. Further, the comparison of goods in the target region images at different time points can be realized, the out-of-stock goods can be determined, and the information such as the type of the out-of-stock goods can be determined by the visual recognition model. On the one hand, the time for labeling all types of goods and training the recognition model can be shortened, thereby saving the cost. The goods can be put on the shelf without labeling and training, thereby improving the real-time performance of putting the goods on the shelf. On the other hand, the open set detection model does not need to accurately determine the type of the goods, thereby improving the recognition speed of the open set detection model. The visual recognition model is used to recognize the out-of-stock goods, thereby improving the speed of determining the out-of-stock goods. On the other hand, since the open set detection model only needs to distinguish between goods and non-goods, the recognition error caused by the fine-grained goods type recognition of the recognition model can be reduced, thereby improving the accuracy of determining the out-of-stock goods.

[0119] In some embodiments, the employing the open set detection model to determine the first goods image corresponding to the goods in the first image and the second goods image corresponding to the goods in the second image respectively comprises:

[0120] employing the open set detection model to recognize the goods in the first image, and employing a predetermined identifier to mark the first goods image corresponding to the recognized goods in the first image;

[0121] employing the open set detection model to recognize the goods in the second image, and employing a predetermined identifier to mark the second goods image corresponding to the recognized goods in the second image;

[0122] the comparing the first goods image corresponding to the goods in the first image and the second goods image corresponding to the goods in the second image comprises:

[0123] comparing the first goods image marked by the predetermined identifier and the second goods image marked by the predetermined identifier.

[0124] In one possible implementation, the information processing device has a display component, and the information processing device can display at least one of the first image, the second image, the first goods image, the second goods image, and the third goods image on the display component.

[0125] In one possible implementation, each goods is marked by the same type of predetermined identifier.

[0126] In one possible implementation, each goods is marked by different types of predetermined identifiers.

[0127] For example, the first goods image in the first image and the second goods image in the second image are both marked by a predetermined identifier such as Figure 5 andFigure 6 The shadowed frame is used to identify.

[0128] In comparing the first image and the second image, the first product image and the second product image marked with the predetermined identification mark can be compared. For example, the parts marked with the shadowed frame in the first image and the second image can be compared. Then, the third product image that is reduced in the second image compared to the first image is determined.

[0129] In one possible implementation, the position of each first product image relative to the shelf in the first image and the position of each second product image relative to the shelf in the second image can be determined first. The out-of-stock position of the shelf with the first product image in the first image and without the first product image in the second image is determined, and the first product image corresponding to the out-of-stock position is determined as the third product image.

[0130] An exemplary, Figure 5 As shown, the first image has products A to G and products 1 to 6, and the open set detection model marks the first image of all products with a shadowed frame. Figure 6 As shown, in the second image, some products are removed, and the open set detection model also marks the second image of all products with a shadowed frame. The information processing device can compare the product images marked with the shadowed frame one by one, so as to determine the third product image that is missing in the second image compared to the first image (the missing third product image is as Figure 7 As shown).

[0131] By marking the first product image and the second product image, the marked first product image and the second product image can be directly compared without comparing other parts in the first image and the second image, so as to improve the comparison efficiency. In addition, by marking the first product image and the second product image, the user can intuitively understand the processing status of the information processing device, so as to improve the exchange with the user and improve the user experience.

[0132] In some embodiments, the comparison of the first product image corresponding to the product in the first image and the second product image corresponding to the product in the second image, the determination of the third product image corresponding to the out-of-stock product that is reduced in the second image compared to the first image, comprises:

[0133] Using a feature matching model, the first product image in the first image and the second product image of the product in the second image are matched, and the first product image that is not matched to the corresponding second product image is determined as the third product image corresponding to the out-of-stock product.

[0134] Here, the feature matching model can be used to extract feature parameters (such as image features, etc.) from the first and second commodity images as matching entities, and the registration of the first and second commodity images is realized by calculating the similarity between the matching entities.

[0135] The feature matching model can accurately match the first and second commodity images, reduce the matching error problem caused by the difference between the first and second images due to different shooting parameters (such as shooting equipment, shooting angle and / or shooting environment) and / or commodity image difference (such as the difference between the first and second images of the same commodity due to commodity movement), and improve the accuracy of determining the third commodity image.

[0136] In some embodiments, the method further comprises: training the visual recognition model capable of recognizing the first classification of the commodity to enable the visual recognition model to recognize the second classification of the commodity, wherein the second classification is a sub-classification of the first classification.

[0137] Here, the visual recognition model can be adjusted to enable the visual recognition model to recognize the second classification in one step. For example, the first classification can be a beverage of brand 1, and the visual recognition model can be further trained to enable the visual recognition model to recognize the flavor of the beverage of brand 1 in one step.

[0138] The visual recognition model can realize the recognition of the second classification based on the recognition of the commodity image and the recognition of the commodity image text.

[0139] Through the recognition of the sub-classification of the first classification of the commodity, the subdivision of the commodity can be realized, the classification of the commodity can be refined, and more accurate out-of-stock commodity information can be provided.

[0140] In some embodiments, the training of the visual recognition model capable of recognizing the first classification of the commodity to enable the visual recognition model to recognize the second classification of the commodity comprises:

[0141] The second training commodity image is input into the visual recognition model, the commodity description containing the second classification description of the commodity in the second training commodity image is output from the visual recognition model, and the visual recognition model is trained.

[0142] In response to the visual recognition model recognizing the second classification of the commodity in the unlabeled commodity image, it is determined that the training of the visual recognition model is completed.

[0143] Specifically, in the training of the visual recognition model, the second classification description can be added to the commodity description as the output of the visual recognition model. Thus, the visual recognition model can realize the fine tuning of the parameters in the training, and further output the second classification.

[0144] Exemplarily, the trained visual recognition model can also be trained by an irregular commodity input visual recognition model, fine-tuned, trained in multiple batches (such as 20 batches), to improve the recognition ability of the visual recognition model.

[0145] The visual recognition model can recognize an image containing another taste of beverage A by inputting a second training commodity image and a text feature description (commodity description), such as an image containing beverage A. For example, the visual recognition model can analyze the text and image feature information in the input commodity image, and output the commodity information of the commodity image as: watermelon-flavored beverage A.

[0146] In this way, the third commodity image of the out-of-stock commodity is recognized in detail by the visual recognition model, and the commodity information (such as the commodity name and the commodity type) of the out-of-stock commodity is obtained, thereby improving the accuracy of commodity recognition.

[0147] In some embodiments, the method further comprises training the open set detection model using a first training commodity image of a first non-standard appearance commodity labeled as a predetermined commodity type.

[0148] In one possible implementation, the commodity types determined by the open set detection model can be used to distinguish commodities and non-commodities. For example, the predetermined commodity types can include “commodity”.

[0149] In one possible implementation, the commodity types determined by the open set detection model can be used to distinguish different types of commodities. For example, the predetermined commodity types can include “beverage”, “dog”, etc.

[0150] The non-standard appearance commodity can be a commodity that breaks through the conventional form of the industry.

[0151] In one possible implementation, the non-standard appearance commodity can include at least one of the following:

[0152] A commodity whose appearance deviates from the typical shape required by the inherent function of the product; (such as biscuits are usually round / square, but designed in the shape of an animal);

[0153] A commodity that directly imitates objects in reality; such as: imitating living beings (tree-shaped pencils, dinosaur erasers), imitating abstract cultural symbols (such as star-shaped lamps).

[0154] Training the open set detection model using the first training commodity image of the first non-standard appearance commodity can enable the open set detection model to recognize non-standard appearance commodities, thereby improving the recognition rate of the open set detection model and further improving the accuracy of determining the out-of-stock commodity.

[0155] In some embodiments, the method further comprises:

[0156] In response to the open set detection model being able to identify the commodity type of the second non-standard appearance commodity in the commodity image, it is determined that the open set detection model completes the training.

[0157] Here, the labeled commodities can be used as the training set to train the open set detection model. For example, for a retail scene, different commodities labeled as "commodities" in the retail scene can be used as the training set to train the open set detection model, so that the open set detection model can identify unlabeled commodity images.

[0158] The open set detection model can be further trained for the actual scene. Here, the targeted training can be training for non-standard commodities. Non-standard commodities can include commodities with irregular shapes and other image features that have a matching degree greater than a threshold with commodity features. For example: a pencil shaped like a potted plant. The targeted training includes (inputting non-standard commodity images in the retail industry into the open set target detection model and defining the label as goods for training iteration until the training is completed (the loss function reaches a preset condition or the training period reaches a preset value) The fine-tuned open set detection model can recognize non-standard commodity images as goods. If the training category is further subdivided into beverages or "dogs", the open set detection model can only frame the beverage commodity or the commodity belonging to the "dog" category when used. Compare the missing "beverage" or "dog" framed in the two frames. Subsequently, the visual recognition model is used to further identify the beverage type and characteristics or the breed and characteristics of the dog.

[0159] The first non-standard appearance commodity image can be input into the open set detection model, and the label can be defined as "commodity" for training iteration until the training is completed (the loss function reaches a preset condition or the training period reaches a preset value), so that the open set detection model after targeted training can identify the second non-standard appearance commodity as "commodity".

[0160] Through the training of the open set detection model, different commodity appearances can be adapted, the accuracy of the open set detection model in identifying commodities can be improved, and the accuracy of determining out-of-stock commodities can be improved.

[0161] In some embodiments, the training of the open set detection model using the first training commodity image of the first non-standard appearance commodity labeled as a predetermined commodity type includes at least one of:

[0162] The first training commodity image is used as input of the open set detection model, the commodity type of the first non-standard appearance commodity is used as label information of the first training commodity image, and the output of the open set detection model is trained.

[0163] The first training commodity image is inputted into the open set detection model, and the position information of the commodity type of the first non-standard appearance commodity in the image description of the first training commodity image is specified, so as to train the output of the open set detection model.

[0164] Here, the first training commodity image can be an image or a part of an image. The position of the first non-standard appearance commodity can be specified during training. For example, the first training commodity image is specified by the position range of the first non-standard appearance commodity in the image.

[0165] When training the open set detection model, the annotation information of the first training commodity image can be directly provided, or the annotation information of the first training commodity image can be specified by indicating the position information of the image description.

[0166] The first training commodity image can be inputted into the open set detection model, and the annotation information can be outputted from the open set detection model, so as to train the open set detection model.

[0167] In one possible implementation, the above two ways of indicating the annotation information can be applied in the training of non-standard appearance commodities, and can also be applied in the training of general commodities.

[0168] For example, the embodiment provides two formats of two training sets in the form of codes:

[0169] Format one:

[0170] Data format of target detection:

[0171] {"filename": "images.jpg",

[0172] "height": 512,

[0173] "width": 769,

[0174] "detection": {

[0175] "instances": [

[0176] {"bbox": [109.4768676992, 346.0190429696, 135.1918335098, 365.3641967616], "label": 2, "category": "goods"},

[0177] {"bbox": [58.612365705900004, 323.2281494016, 242.6005859067, 451.4166870016], "label": 8, "category": "car"}

[0178] ]}}

[0179] In the first format, the training image of the goods is provided: images.jpg; and the positions of the two first non-standard appearance goods in the training image of the goods are specified through "bbox", and the types of the two first non-standard appearance goods are given "category": "goods" and "car".

[0180] Format two:

[0181] The second format is data format:

[0182] {"filename": "2405116.jpg",

[0183] "height": 375,

[0184] "width": 500,

[0185] "grounding":

[0186] {"caption": "Two surfers walking down the shore. sand on the beach.",

[0187] "regions": [{"bbox": [206, 156, 282, 248], "phrase": "Two surfers", "tokens_positive": [[0, 3], [4, 11]]},

[0188] {"bbox": [303, 338, 443, 343], "phrase": "sand", "tokens_positive": [[36, 40]]},

[0189] {"bbox": [[327, 223, 421, 282], [300, 200, 400, 210]], "phrase": "beach", "tokens_positive": [[48, 53]]}

[0190] ]}}

[0191] In the format two, a training commodity image 2405116.jpg is provided, and the position information of three to-be-identified objects in the training image is specified through "regions" and "bbox", respectively, and the position information of each to-be-identified object in the training image description "Two surfers walking down the shore. sand on the beach." is indicated through "tokens_positive". For example, "tokens_positive": [[36, 40]], which means that the annotation of the to-be-identified object is the 36th to 40th character in the training image description.

[0192] In some embodiments, after determining the commodity information of the out-of-stock commodity corresponding to the third commodity image, the method further comprises:

[0193] sending an out-of-stock indication indicating the commodity information corresponding to the out-of-stock commodity.

[0194] Here, the out-of-stock indication can be used to instruct the user to replenish the out-of-stock commodity. The out-of-stock indication can also indicate the quantity of the out-of-stock commodity, etc. determined based on the commodity information of the out-of-stock commodity.

[0195] The sending of the out-of-stock indication indicating the commodity information corresponding to the out-of-stock commodity can include at least one of the following:

[0196] displaying the out-of-stock indication on the display component of the information processing device to indicate the commodity information of the out-of-stock commodity;

[0197] sending the out-of-stock indication to a user terminal (such as a mobile phone, etc.) to indicate the commodity information of the out-of-stock commodity.

[0198] The following provides a plurality of specific examples in combination with any of the above embodiments:

[0199] This example discloses a shelf commodity out-of-stock detection method, as shown in Figure 8 The method comprises the following steps:

[0200] Step 801, model training is performed on the open set detection model. Based on a general open set detection model, a labeled commodity data set is used for training to obtain an open set detection model adapted to a commodity task;

[0201] For example, the real-time open set detection model Grounding-DINO can be selected for fine-tuning, and the pre-training model of the Grounding-DINO detection model is selected. The pre-training model of the Grounding-DINO is trained on the COCO, RefC, O365V2, GoldG, GRIT, Open-Images, and V3Det data sets.

[0202] The data categories of the public dataset do not completely adapt to the retail scene, so a labeled retail scene dataset is used for targeted training. Targeted training includes: inputting non-standard commodity (non-standard appearance) images in the retail industry into the open set target detection model, defining the label as goods for training iteration, and training until completion (such as the loss function reaching the preset condition or the training period reaching the preset value). The fine-tuned open set detection model can recognize non-standard commodity images as goods.

[0203] If the training category is further subdivided into beverages or "dogs", then the open set detection model can only frame the beverage category or the "dog" category when used, and compare the missing "beverage" or "dog" framed in the two frames. Further use of a visual large model to further identify beverage types and characteristics or dog breeds and characteristics.

[0204] Step 802, using an open set detection model to detect an initial image. Use the trained open set detection model to detect all goods on the shelf to generate a complete goods inventory reference image, as shown in Figure 5 .

[0205] Specifically, the frame image in the video stream of the oblique camera device or the image taken by the mobile phone and the like is obtained, and the trained open set detection model is used to obtain the detection frame and category (category is goods) of the goods. The detection result is used as the category reference image of the shelf.

[0206] Step 803, using an open set detection model to detect real-time images. Use the trained open set detection model to detect all goods on the shelf in real time to generate real-time images, as shown in Figure 6 .

[0207] Step 804, using a feature matching algorithm to compare the distribution of goods in the initial image and the real-time image, and marking the goods that fail to match as out-of-stock goods. Goods that fail to match are shown in Figure 7 .

[0208] Specifically:

[0209] 1) A specific inspection time uses a mobile phone or the like to take a picture of the shelf or uses an oblique camera to obtain a real-time image frame, and uses a trained open set detection model to obtain the detection frame and category of the goods.

[0210] 2) Use a feature matching method to match the target in the current image with the target in the reference image, and screen out the out-of-stock target.

[0211] Step 805, using a visual large model to identify the out-of-stock goods in detail to obtain the name and category information of the goods.

[0212] Specifically, the out-of-stock target image is input into the visual large model in turn, and the recognized target name is output. The visual large model can be selected from VITA-1.5 and other open source models.

[0213] The example also discloses a shelf commodity out-of-stock detection system, Figure 9 The operation schematic diagram of the shelf commodity out-of-stock detection system of the example includes steps 901 to 906.

[0214] Step 901, set up an image acquisition device to acquire original commodity images (i.e. initial images, first images) and real-time commodity images (i.e. second images). The oblique camera device installed on a supermarket or a store acquires commodities on the original shelf and commodities on the real-time shelf.

[0215] Step 902, open set detection model loading, preloading a trained open set detection model adapted to the commodity task.

[0216] Before step 902, it also includes training the open set detection model, specifically including:

[0217] Step 9021: fine-tune on the basis of a general open set detection model, and use its disclosed pre-training model to train on a public data set, and then use a labeled retail scene data set for targeted training (supplement retail industry data as product data, special commodities such as irregularly shaped commodities such as potted pencils), so that the open set detection model better adapts to the retail scene.

[0218] Step 903, detect the original commodity image. Initially, the open set detection model is used to detect the acquired original commodity image of the shelf, to obtain the detection frame and category of the commodity, and the detection result is used as the reference image of the shelf.

[0219] Step 904, detect the real-time commodity image. Use a shooting device or a camera to obtain a real-time commodity image of the shelf, and use the trained open set detection model to obtain the detection frame and category of the commodity.

[0220] Step 905, feature matching. Use a feature matching method to match the images and screen out the out-of-stock target.

[0221] Step 906, use a visual large model to perform fine-grained identification on the out-of-stock commodity to obtain commodity information (commodity name and commodity category).

[0222] Specifically: the current open source visual large model (VITA1.5) can be selected for fine-tuning, and the pre-trained model disclosed by the visual large model is used to train on a public data set. The visual large model is trained for irregular commodity input, and the parameters are fine-tuned. After training for about 20 batches, it can be determined whether the training is completed. Input the image and text feature description of the commodity, such as inputting the image containing the original flavor beverage A. The visual large model can identify the image containing other flavor beverages A. The visual large model can analyze the text and image feature information in the input image and output the commodity information in the image as: beverage A with honey peach flavor. The image of the out-of-stock commodity is input into the visual large model in turn, and the recognized commodity name can be output.

[0223] Figure 10 The embodiment of the application provides a commodity information determination device 100, and the device comprises a processing module 110, which is used for:

[0224] A first commodity image corresponding to the commodity in the first image and a second commodity image corresponding to the commodity in the second image are determined by using an open set detection model, wherein the first image is an image of a target region at a first time, and the second image is an image of the target region at a second time, wherein the first time is before the second time;

[0225] The first commodity image corresponding to the commodity in the first image and the second commodity image corresponding to the commodity in the second image are compared, and a third commodity image corresponding to a lack-of-stock commodity that is reduced in the second image compared with the first image is determined;

[0226] The third commodity image is identified by using a visual recognition model, and commodity information of the lack-of-stock commodity corresponding to the third commodity image is determined.

[0227] In some embodiments, the processing module is specifically used for:

[0228] The commodities in the first image are identified by using the open set detection model, and the first commodity image corresponding to the identified commodities in the first image is marked by using a predetermined identifier;

[0229] The commodities in the second image are identified by using the open set detection model, and the second commodity image corresponding to the identified commodities in the second image is marked by using a predetermined identifier;

[0230] The first commodity image marked by using the predetermined identifier and the second commodity image marked by using the predetermined identifier are compared.

[0231] In some embodiments, the device further comprises a training module 120,

[0232] The training module is configured to train the open-set detection model by using a first training commodity image of a first non-standard appearance commodity of a predetermined commodity type.

[0233] In some embodiments, the training module is specifically configured to at least one of:

[0234] training the open-set detection model by taking the first training commodity image as input and the commodity type of the first non-standard appearance commodity as annotation information of the first training commodity image, and training an output of the open-set detection model;

[0235] training the open-set detection model by taking the first training commodity image as input and specifying position information of the commodity type of the first non-standard appearance commodity in an image description of the first training commodity image, and training an output of the open-set detection model.

[0236] In some embodiments, the training module is further configured to determine that the open-set detection model is trained in response to the open-set detection model being able to identify the commodity type of a second non-standard appearance commodity in a commodity image.

[0237] In some embodiments, the apparatus further comprises a training module,

[0238] The training module is configured to train a visual recognition model capable of identifying a first classification of commodities to enable the visual recognition model to identify a second classification of commodities, wherein the second classification is a sub-classification of the first classification.

[0239] In some embodiments, the training module is specifically configured to:

[0240] training the visual recognition model by taking a second training commodity image as input and a commodity description containing a second classification description of the commodity in the second training commodity image as output, and training the visual recognition model;

[0241] determining that the visual recognition model is trained in response to the visual recognition model identifying the second classification of commodities in an unlabeled commodity image.

[0242] It should be understood that the division of units or modules in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the units or modules in the device can be implemented by a processor calling software: for example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of the units or modules in the above device. The processor can be, for example, a general-purpose processor, such as a Central Processing Unit (CPU) or a microprocessor, and the memory can be internal or external to the device. Alternatively, the units or modules in the device can be implemented in the form of hardware circuits. The functionality of some or all of the units or modules can be achieved through the design of these hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functionality of some or all of the units or modules is achieved through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a programmable logic device (PLD), such as a field-programmable gate array (FPGA), which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby achieving the functionality of some or all of the units or modules. All units or modules of the above device can be implemented entirely through processor-called software, entirely through hardware circuits, or partially through processor-called software with the remaining parts implemented through hardware circuits.

[0243] In the embodiments of the present disclosure, the processor is a circuit with data processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), and the like. In another implementation, the processor can implement certain functions through a logical relationship of hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable. For example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the configuration of the hardware circuit. It can be understood that the processor loads instructions to implement the functions of the above part or all units or modules. In addition, the hardware circuit can also be designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), and the like.

[0244] Figure 11 FIG. 9 is a structural schematic diagram of an electronic device 9100 provided by the embodiments of the present disclosure. The electronic device 9100 can be a computer device, a terminal, a chip, a chip system, a processor, or the like supporting the implementation of any of the above methods. The electronic device 9100 can be used to implement the commodity information determination method described in the above method embodiments, and specific implementation can be referred to the description in the above method embodiments.

[0245] As shown in FIG. 9, the electronic device 9100 includes one or more processors 9101. The processor 9101 can be a general-purpose processor or a special-purpose processor, etc. The processor 9101 is configured to invoke instructions to enable the electronic device 9100 to perform any of the above commodity information determination methods. Figure 10

[0246] In some embodiments, the electronic device 9100 further includes one or more memories 9102 for storing instructions. Optionally, all or part of the memory 9102 can also be outside the electronic device 9100.

[0247] ​In some embodiments, the electronic device 9100 further includes one or more transceivers 9103. When the electronic device 9100 includes one or more transceivers 9103, the steps of sending, receiving, and / or acquiring, etc. in the above methods are performed by the transceiver 9103, and other steps are performed by the processor 9101.

[0248] In some embodiments, the steps of acquiring, etc. in the above methods can also be performed by the processor 9101, for example, acquiring information from the memory 9102, etc.

[0249] In some embodiments, the transceiver can include a receiver and a transmitter, which can be separate or integrated together. Optionally, the terms of transceiver, transceiving unit, transceiver, transceiving circuit, etc. can be replaced by each other, the terms of transmitter, transmitting unit, transmitter, transmitting circuit, etc. can be replaced by each other, and the terms of receiver, receiving unit, receiver, receiving circuit, etc. can be replaced by each other.

[0250] Optionally, the electronic device 9100 further includes one or more interface circuits 9104, which are connected with the memory 9102, and can be used to receive signals from the memory 9102 or other devices, and can be used to send signals to the memory 9102 or other devices. For example, the interface circuit 9104 can read the instructions stored in the memory 9102 and send the instructions to the processor 9101.

[0251] The electronic device 9100 described in the above embodiments can be a network device or a terminal, but the scope of the electronic device 9100 described in the present disclosure is not limited thereto, and the structure of the electronic device 9100 can not be limited by Figure 10 The electronic device can be a stand-alone device or can be part of a larger device. For example, the electronic device can be: (1) a stand-alone integrated circuit (IC), or a chip, or a chip system or subsystem; (2) a set of one or more ICs, which can optionally also include storage components for storing data, programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal device, a smart terminal device, a cellular phone, a wireless device, a handset, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, etc.; (6) others, etc.

[0252] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program commands related to hardware, and the foregoing programs can be stored in a storage medium, including mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic discs or optical discs, and various storage media that can store program codes.

[0253] Alternatively, the integrated units described above in the present application, if implemented in the form of software function modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products, and the computer software products are stored in a storage medium, including a number of commands for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROM, RAM, magnetic discs or optical discs, and various storage media that can store program codes.

[0254] It should be understood that the above embodiments are all exemplary and are not intended to include all possible implementations. Various modifications and changes can also be made on the basis of the above embodiments without departing from the scope of the present disclosure. Similarly, any combination of the technical features of the above embodiments can also be made to form additional embodiments of the present application that can not have been explicitly described. Therefore, the above embodiments only express several implementation manners of the present application, and do not limit the protection scope of the patent of the present application.

Claims

1. A commodity information determination method characterized by comprising: The method comprises: adopting an open set detection model to determine a first commodity image corresponding to a commodity in a first image and a second commodity image corresponding to a commodity in a second image, respectively, wherein the first image is an image of a target region at a first time, the second image is an image of the target region at a second time, and the first time is before the second time; wherein the open set detection model is used to distinguish commodities and non-commodities in the first image and the second image; comparing the first commodity image corresponding to the commodity in the first image and the second commodity image corresponding to the commodity in the second image to determine a third commodity image corresponding to a shortage commodity that is reduced in the second image compared with the first image; one third commodity image corresponds to one shortage commodity; adopting a visual recognition model to identify the third commodity image to determine commodity information of the shortage commodity corresponding to the third commodity image; The method comprises: adopting the open set detection model to identify commodities in the first image, and using a predetermined identifier to mark the first commodity image corresponding to the identified commodities in the first image; one commodity in the first image corresponds to one first commodity image; adopting the open set detection model to identify commodities in the second image, and using a predetermined identifier to mark the second commodity image corresponding to the identified commodities in the second image; one commodity in the second image corresponds to one second commodity image; The method comprises: comparing the first commodity image marked with the predetermined identifier and the second commodity image marked with the predetermined identifier; The method further comprises: training the open set detection model using a first training commodity image of a first non-standard appearance commodity labeled as a commodity; the first non-standard appearance commodity includes at least one of the following: a commodity whose appearance deviates from the typical shape required by the inherent function of the product; a commodity that directly imitates a real object.

2. The merchandise information determination method according to claim 1, characterized by, The method further comprises: The method further comprises: The method further comprises:

3. The merchandise information determination method according to claim 1, characterized by, The method further comprises: The method further comprises:

4. The merchandise information determination method according to claim 1, characterized by, ​ The visual recognition model capable of recognizing the first classification of the commodity is trained to enable the visual recognition model to recognize a second classification of the commodity, wherein the second classification is a sub-classification of the first classification.

5. The merchandise information determination method according to claim 4, characterized by, The visual recognition model capable of recognizing the first classification of the commodity is trained to enable the visual recognition model to recognize a second classification of the commodity; Comprise: The second training commodity image is taken as an input of the visual recognition model, and a commodity description containing a second classification description of the commodity in the second training commodity image is taken as an output of the visual recognition model, and the visual recognition model is trained. In response to the visual recognition model recognizing the second classification of the commodity in the unlabeled commodity image, it is determined that the training of the visual recognition model is completed.

6. A commodity information determination device characterized by comprising: The device comprises a processing module, wherein the processing module is configured to: A first commodity image corresponding to a commodity in a first image and a second commodity image corresponding to a commodity in a second image are determined by using an open set detection model, wherein the first image is an image of a target region at a first time, and the second image is an image of the target region at a second time, wherein the first time is before the second time; wherein the open set detection model is used to distinguish commodities and non-commodities in the first image and the second image; The first commodity image corresponding to the commodity in the first image and the second commodity image corresponding to the commodity in the second image are compared to determine a third commodity image corresponding to a commodity out of stock in the second image compared with the first image, and one third commodity image corresponds to one commodity out of stock; The third commodity image is identified by using a visual recognition model to determine commodity information of the commodity out of stock corresponding to the third commodity image; The processing module is specifically configured to: The commodities in the first image are identified by using the open set detection model, and the first commodity images corresponding to the identified commodities in the first image are marked by using a predetermined identifier; one commodity in the first image corresponds to one first commodity image; The commodities in the second image are identified by using the open set detection model, and the second commodity images corresponding to the identified commodities in the second image are marked by using a predetermined identifier; one commodity in the second image corresponds to one second commodity image; The first commodity images marked by using the predetermined identifier and the second commodity images marked by using the predetermined identifier are compared; The device further comprises a training module, The training module is configured to train the open set detection model by using a first training commodity image of a first non-standard appearance commodity labeled as a predetermined commodity type; the first non-standard appearance commodity comprises at least one of the following: a commodity whose appearance deviates from a typical shape required by a product inherent function; a commodity directly imitating a real object.

7. An electronic device, comprising: The electronic device comprises: One or more processors; The processor is configured to call instructions to enable the electronic device to perform the commodity information determination method in any one of claims 1 to 5.

8. A storage medium, characterized by The storage medium stores instructions which, when executed on the electronic device, cause the electronic device to perform the commodity information determination method of any one of claims 1 to 5.

Images