Image-based commodity retrieval method, device, equipment and storage medium

Abstract

The present application discloses an image-based commodity retrieval method, device, equipment and storage medium, the method comprising: in response to a retrieval trigger signal uploaded by a store terminal device, extracting a commodity image in the retrieval trigger signal; inputting the commodity image into a commodity recognition model so that the commodity recognition model outputs the commodity SKU value; querying multiple packaging SKU values ​​corresponding to the commodity SKU value and the purchase price corresponding to the packaging SKU value according to the SKU value; feeding back the packaging SKU value and the purchase price to the store terminal device; in response to a selection trigger signal uploaded by the store terminal device, generating shopping cart item data, the shopping cart item data including the packaging SKU value, the purchase price and the recommended purchase quantity. The present application is beneficial in that the method of image retrieval can effectively retrieve commodities and generate shopping cart items that can be directly ordered.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image-based product retrieval method. Background Art

[0002] In related technologies, orders from multiple stores are aggregated through an Internet platform, and then unified procurement and logistics pickup are carried out from suppliers. The carrier vehicles then deliver the goods to the corresponding stores according to the purchase orders, thereby reducing the warehousing costs of stores such as convenience stores and improving procurement flexibility.

[0003] Nowadays, there are many stores that need to purchase goods. If you use fields other than search, it will take a long time and be inconvenient to operate. If you use the collection or shopping cart method, it will lead to too many product items and lose the purpose of saving operations. In addition, there will be changes in the warehouse, and the product items in the collection and shopping cart will easily become invalid. Summary of the Invention

[0004] The content of this application is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this application is not intended to identify key features or essential features of the technical solution for which protection is sought, nor is it intended to limit the scope of the technical solution for which protection is sought.

[0005] Some embodiments of the present application propose image-based product retrieval methods, devices, electronic devices, and storage media to solve the technical problems mentioned in the above background technology section.

[0006] As a first aspect of the present application, some embodiments of the present application provide an image-based product retrieval method, comprising the following steps: in response to a retrieval trigger signal uploaded by a store terminal device, extracting a product image in the retrieval trigger signal; inputting the product image into a product recognition model so that the product recognition model outputs the product SKU value; querying multiple packaging SKU values ​​corresponding to the product SKU value and the purchase price corresponding to the packaging SKU value based on the SKU value; feeding back the packaging SKU value and the purchase price to the store terminal device; and generating shopping cart entry data in response to a selection trigger signal uploaded by the store terminal device, the shopping cart entry data including the packaging SKU value, the purchase price and the recommended purchase quantity.

[0007] As the second aspect of the present application, some embodiments of the present application provide an image-based product retrieval device, including: an extraction module for extracting the collection location data of the product image in the retrieval starting signal, and the device coding data of the store terminal device; an identification module for inputting the product image into the product identification model so that the product identification model outputs the product SKU value; a query module for querying the multiple packaging SKU values ​​corresponding to the product SKU value and the purchase price corresponding to the packaging SKU value as the current purchase price based on the SKU value; a feedback module for feeding back the packaging SKU value and the purchase price to the store terminal device; an ordering module for generating shopping cart item data output by placing an order based on the selection trigger signal uploaded by the store terminal device.

[0008] As the third aspect of the present application, some embodiments of the present application provide an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation method of the above-mentioned first aspect.

[0009] As a fourth aspect of the present application, some embodiments of the present application provide a storage medium on which a computer program is stored, wherein when the program is executed by a processor, the method described in any implementation of the first aspect above is implemented.

[0010] The benefit of this application is that it can effectively retrieve products through the image retrieval method and generate shopping cart items that can be directly ordered.

[0011] Some embodiments of the present application may produce the following specific beneficial effects: by associating pictures with packaging SKU values, operation steps are saved and even if the product entry is invalid, it has no impact on the product itself; the product recognition model based on store classification is helpful for model training and improving the confidence of the result output; the remote association retrieval method helps store users obtain new products. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings constituting a part of this application are used to provide a further understanding of this application and make other features, purposes and advantages of this application more apparent. The drawings and descriptions of the exemplary embodiments of this application are used to explain this application and do not constitute an improper limitation on this application.

[0013] In addition, throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the elements and components are not necessarily drawn to scale.

[0014] Figure 1 This is a schematic diagram of a "group order" purchasing model for stores in related technologies;

[0015] Figure 2 is a flowchart of an image-based product retrieval method according to an embodiment of the present application;

[0016] Figure 3 This is a flowchart of some specific steps in an image-based product retrieval method according to an embodiment of the present application;

[0017] Figure 4 This is a flowchart of another part of the specific steps in the image-based product retrieval method according to an embodiment of the present application;

[0018] Figure 5 This is a flowchart of another part of the specific steps in the image-based product retrieval method according to an embodiment of the present application;

[0019] Figure 6 is a flowchart of another part of the specific steps in the image-based product retrieval method according to an embodiment of the present application;

[0020] Figure 7 is a structural diagram of an image-based product retrieval device according to an embodiment of the present application;

[0021] Figure 8 It is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0023] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0024] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "multiple" mentioned in this application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0028] Reference Figure 2 As shown, an image-based product retrieval method according to an embodiment of the present application includes the following steps:

[0029] Step S01: In response to a search trigger signal uploaded by a store terminal device, extract product images in the search trigger signal.

[0030] Step S02: Input the product image into a product recognition model so that the product recognition model outputs the product SKU value.

[0031] Step S03: Query multiple packaging SKU values ​​corresponding to the product SKU value and the purchase price corresponding to the packaging SKU value according to the SKU value.

[0032] Step S04: Feedback the packaging SKU value and purchase price to the store terminal device.

[0033] Step S05: In response to the selection trigger signal uploaded by the store terminal device, shopping cart entry data is generated. The shopping cart entry data includes the package SKU value, purchase price, and recommended purchase quantity. The purchase price here refers to the current purchase price on the e-commerce platform, which is updated in real time. This avoids the problem of expired purchase prices in "group purchases" caused by expired collections and shopping carts. This solution allows users to place orders instantly, improving ordering convenience.

[0034] As a specific solution, a store terminal device has a touch screen and a camera, and can be constructed as a smartphone. The store terminal device's retrieval trigger signal can be generated by the operation or result of collecting product images. There are two specific application scenarios: one is for store users to collect product images from their own store to enable automatic identification and ordering, and the other is for store users to collect product images from other stores or locations to expand their product categories.

[0035] In order to be suitable for these two application scenarios, such as Figure 3As shown, as a preferred solution, the above-mentioned image-based product retrieval method also includes the following steps: extracting the acquisition location data of the product image in the retrieval trigger signal; extracting the device code data of the store terminal device in the retrieval trigger signal; obtaining the corresponding store registration location based on the device code data; judging whether the acquisition location data is consistent with the store registration location, if so, querying the store grouping based on the store registration location, if not, obtaining the store registration location of other stores nearest to the store according to the location in the acquisition location data, and querying the store grouping location based on the store registration location of other stores; determining and selecting the product recognition model based on the store grouping location.

[0036] This solution can be used to determine whether the user is in the store or elsewhere, and select different product recognition models based on the location. As we all know, training a neural network model requires the training set data to have certain stable characteristics and a sufficient amount of data.

[0037] If only the historical data of one store terminal device is used to generate training set data, there will be a problem of insufficient data volume. If the historical data of all store terminal devices in the system are used for training, the model training will not converge because the characteristics of the product image data are too scattered, or the model accuracy will be poor after training.

[0038] In view of the above, this application provides a technical solution to the above problem by grouping stores. Through research, it is found that stores of the same size have certain similarities in product placement, and if they are geographically close, they also have certain similarities in purchasing behavior.

[0039] Therefore, in order to facilitate the training of the product recognition model and enable it to better identify product images, certain store groups are formed based on the above characteristics, so that stores within the store group can learn from each other's historical data, that is, to obtain sufficient training data while making the training data itself have a certain relevance, which is convenient for model training.

[0040] Especially when searching for product images outside of one's own store, the product recognition model is obtained by searching the store group to which the nearest store belongs, thereby improving the search efficiency and accuracy.

[0041] As a preferred solution, Figure 4 As shown, the above-mentioned image-based product retrieval method also includes the following steps: dividing the store terminal devices into different store groups at least according to the store registration location of the store terminal devices; configuring a product recognition model for each store group, and the product recognition model is trained by the historical product image data and historical product SKU values ​​uploaded by the store terminal devices in the store group as input data and output data respectively.

[0042] As a preferred solution, the collection location data of the product image in the retrieval trigger signal and the device code data of the store terminal device are extracted, and the acquired device code data is compared with the corresponding store registration location to determine whether the collection location data is consistent with the store registration location. If it is consistent, the store grouping is queried based on the store registration location. If it is not consistent, the store registration location of the nearest other store is obtained based on the location in the collection location data, and the store grouping location is queried based on the store registration location of other stores.

[0043] In view of the above, if Figure 4 As shown, as a further preferred solution, the above-mentioned image-based product retrieval method also includes the following steps: dividing the store terminal devices into different store groups at least according to the store registration location of the store terminal devices; configuring a product recognition model for each store group, and the product recognition model is trained by the historical product image data and historical product SKU values ​​uploaded by the store terminal devices in the store group as input data and output data respectively.

[0044] like Figure 5 As shown, as a specific solution, the specific method of store grouping includes the following steps:

[0045] Obtain historical purchase order data from store terminal devices.

[0046] Calculate the average order value of store terminal devices based on historical purchase order data.

[0047] A three-dimensional coordinate system is established with average order value as the third dimension.

[0048] Get the coordinate values ​​of the store terminal device in the above three-dimensional coordinate system. The coordinates of the other two dimensions of the three-dimensional coordinate system are the two-dimensional coordinates of the store registration location of the store terminal device in the plane map.

[0049] K-Means clustering operation is performed based on the coordinate values ​​of the store terminal devices in the three-dimensional coordinate system.

[0050] The stores are divided into groups according to the K-Means clustering operation results.

[0051] More specifically, the historical orders referred to here are the store's purchase orders. The average order value is used as the third dimension to establish a three-dimensional coordinate system. The other two dimensions are used to establish a two-dimensional coordinate system based on the geographic location. The coordinate values ​​of the store in the two-dimensional coordinate system are combined with the average order value of the third dimension to form this three-dimensional coordinate system. For example, assuming that the coordinate axes of the three-dimensional coordinate system are X, Y, and Z, the coordinates of the store on the X and Y axes are the store's location coordinates on the plane map, and the Z-axis coordinate is the store's average order value. The average order value is the average value of the order values ​​of all purchase orders of the store during the observation period. Obtain the coordinate values ​​of the store in the above three-dimensional coordinate system. The location coordinates can be obtained through maps and positioning data, and the average order value is obtained by calculating historical data. As a preferred option, the observation period is quarterly or annual. Such a longer time period can more stably reflect the characteristics of the store.

[0052] A store group is essentially a collection of stores. From the perspective of data representation, it can be expressed as the store's ID. Of course, it can also be expressed as the store's name, but the text itself is not suitable for data processing. It is preferred to use the store's unique store ID code, or the store user's unique account ID. The two have the same function.

[0053] Continue to refer to Figure 6 As shown, step S02 specifically includes the following steps: the product recognition model outputs several detection frames and the product SKU values ​​corresponding to the detection frames; determines whether the confidence of each detection frame and its product SKU value meets the confidence threshold, and if so, retains the product SKU value; if not, sends the detection frame to the store terminal device to prompt the user to re-capture the image; removes duplicate product SKU values, and only saves one detection frame for the same SKU value for confidence threshold judgment.

[0054] When collecting product images, there may be multiple images of the same product. In this case, only one of them needs to be retained. As a preferred solution, the corresponding confidence levels of multiple detection frames of the same product can be generated separately, and the one with the highest confidence level is used for confidence judgment.

[0055] In addition, it should be noted that the product recognition model of the present application can be a convolutional neural network model, which can realize the functions of detection frame generation and image classification within the detection frame (classification corresponds to product SKU). Such a convolutional neural network model is a technical solution well known to general technicians in the field of image recognition. If constructing this network is achievable by technical personnel in this field according to functional requirements, it is not the focus of this application and will not be elaborated here.

[0056] like Figure 7As shown, the image-based product retrieval device as an implementation method specifically includes: an extraction module, which is used to extract the collection position data of the product image in the retrieval starting signal, and the device coding data of the store terminal device; an identification module, which is used to input the product image into the product identification model so that the product identification model outputs the product SKU value; a query module, which is used to query the multiple packaging SKU values ​​corresponding to the product SKU value and the purchase price corresponding to the packaging SKU value as the current purchase price according to the SKU value; a feedback module, which is used to feed back the packaging SKU value and the purchase price to the store terminal device; an ordering module, which is used to generate shopping cart item data output by placing an order based on the selection trigger signal uploaded by the store terminal device.

[0057] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0058] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 8 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0059] In particular, according to some embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network via the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of some embodiments of the present application are performed.

[0060] It should be noted that the computer-readable medium in some embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0061] In some embodiments of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0062] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an adhoc peer-to-peer network), as well as any currently known or future developed network.

[0063] Computer program code for performing the operations of some embodiments of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0064] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function.

[0065] It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures.

[0066] For example, two blocks shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow charts, and combinations of blocks in the block diagrams and / or flow charts, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or may be implemented using a combination of dedicated hardware and computer instructions.

[0067] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0068] The above descriptions are merely some preferred embodiments of the present application and illustrate the technical principles employed. Those skilled in the art should understand that the scope of the invention described in the embodiments of the present application is not limited to technical solutions formed by specific combinations of the aforementioned technical features, but also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present application.

Claims

1. A commodity retrieval method based on an image, characterized in that: The image-based commodity retrieval method comprises the following steps: In response to a search trigger signal uploaded by a store terminal device, extracting a product image in the search trigger signal; Inputting the product image into a product recognition model so that the product recognition model outputs a product SKU value; According to the SKU value, query multiple packaging SKU values ​​corresponding to the product SKU value and the purchase price corresponding to the packaging SKU value; Feedback the packaging SKU value and the purchase price to the store terminal device; In response to a selection trigger signal uploaded by the store terminal device, generating shopping cart item data, the shopping cart item data including the package SKU value, the purchase price and the recommended purchase quantity; The image-based commodity retrieval method further includes the following steps: Extracting the acquisition location data of the product image in the retrieval trigger signal; extracting the device code data of the store terminal device in the retrieval trigger signal; Acquire the corresponding store registration location according to the device coding data; Determine whether the collected location data matches the store registration location, if yes, query the store grouping according to the store registration location, if no, obtain the store registration location of other stores nearest to the location in the collected location data, and query the store grouping location according to the store registration location of other stores; Determine and select the commodity recognition model according to the store grouping location; The image-based commodity retrieval method further includes the following steps: dividing the store terminal devices into different store groups at least according to the store registration locations of the store terminal devices; A product recognition model is configured for each store group, wherein the product recognition model is trained using historical product image data and historical product SKU values ​​uploaded by the store terminal devices in the store group as input data and output data respectively; The dividing the store terminal devices into different store groups at least according to the store registration locations of the store terminal devices comprises the following steps: Obtain historical purchase order data from store terminal devices; Calculate the average order value of the store terminal devices based on the historical purchase order data; Use the average order value as the third dimension to establish a three-dimensional coordinate system; Obtaining the coordinate value of the store terminal device in the above three-dimensional coordinate system, the coordinates of the other two dimensions of the three-dimensional coordinate system are the two-dimensional coordinates of the store registration position of the store terminal device in the plane map; Perform K-Means clustering operation based on the coordinate values ​​of the store terminal equipment in the three-dimensional coordinate system; Divide the stores into groups according to the K-Means clustering operation result; The step of inputting the product image into a product recognition model so that the product recognition model outputs the product SKU value comprises the following steps: The commodity recognition model outputs a plurality of detection frames and the commodity SKU values ​​corresponding to the detection frames; Determine whether the confidence of each detection frame and its product SKU value meets a confidence threshold, if yes, retain the product SKU value, if not, send the detection frame to the store terminal device to prompt the user to re-capture the image; The step of inputting the product image into a product recognition model so that the product recognition model outputs the product SKU value comprises the following steps: Duplicate product SKU values ​​are removed, and only one detection box is saved for the same SKU value to perform the confidence threshold judgment; The purchase price among the purchase prices corresponding to the multiple packaging SKU values ​​and the packaging SKU values ​​queried according to the SKU value is the current purchase price.

2. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the processors implement the method as claimed in claim 1.

3. A storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to claim 1 is implemented.

Images