Method, apparatus, device and medium for identifying false image of store commodity display

Abstract

The present invention discloses a method, apparatus, device and medium for identifying image forgery of store product displays. The method includes: obtaining the location information of multiple stores and the corresponding product display images of each store; grouping the multiple stores according to the location information of the stores, and for each group, performing the following steps: extracting features from the product display images of each store in the group to obtain first image features; using a preset first autoencoder model to process the SKU information of the product display images to obtain second image features; performing fusion processing on the first image features and the second image features to obtain a fusion feature matrix; calculating the similarity between the pairwise feature vectors of the fusion feature matrix, and identifying the stores with forged product display images according to the calculated similarity results. The above method not only combines image feature learning for apparent similarity, but also uses semantic annotation learning for further fusion, thereby improving the accuracy of image forgery identification.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a method, device, equipment and medium for identifying image forgery of store product display. Background Art

[0002] In the fast-moving consumer goods industry, when accounting for the investment of product display activity expenses, business personnel often need to take pictures of the product display in stores for collection, and the background conducts activity expense accounting based on the collected product display image information. However, some business personnel may over-report the product display in stores, or use images of stores with display activities in neighboring stores to replace stores without display activities for false reporting. Therefore, it is usually difficult to quickly determine whether there is forgery in the huge amount of store display photo data.

[0003] Currently, manufacturers mainly identify forgery by dispatching supervisors to stores for manual spot checks. However, this method not only has a high cost but also low efficiency. On the other hand, existing image recognition technologies can obtain good discrimination results for the recognition of the same pictures, but for similar product display pictures, the judgment accuracy is often low. Summary of the Invention

[0004] Aiming at the above technical problems, the present invention provides a method, device, equipment and medium for identifying image forgery of store product display, which can accurately and quickly identify stores with image forgery of product display from the huge amount of store display photo data.

[0005] In a first aspect, the present invention provides a method for identifying image forgery of store product display, including:

[0006] Obtain the location information of multiple stores and the product display image corresponding to each store;

[0007] Group the multiple stores according to the location information of the stores. For each group, perform the following steps:

[0008] Extract features from the product display images of the stores in the group to obtain first image features;

[0009] Process the SKU information of the product display image by using a preset first autoencoder model to obtain second image features;

[0010] Perform fusion processing on the first image features and the second image features to obtain a fusion feature matrix;

[0011] Calculate the similarity between the pairwise feature vectors of the fusion feature matrix, and identify stores with image forgery of product display according to the calculated similarity results.

[0012] Optionally, based on the calculated similarity result, identify the stores where there is fraud in the product display images, specifically as follows:

[0013] Determine whether the calculated similarity is greater than a preset threshold;

[0014] If there is a similarity greater than the preset threshold, based on the index of the fusion feature matrix, determine the corresponding stores where there is fraud in the product display images.

[0015] Optionally, the process of fusing the first image feature and the second image feature to obtain a fusion feature matrix is specifically as follows:

[0016] Horizontally splice the first image feature and the second image feature to obtain a spliced feature matrix;

[0017] Use a preset second autoencoder model to process the spliced feature matrix to obtain a fusion feature matrix.

[0018] Optionally, use ResNet101 as the backbone network to extract image features from the product display images of each store in the group to obtain the first image feature.

[0019] Optionally, the preset first autoencoder model is specifically a variational autoencoder model.

[0020] In a second aspect, the present invention also provides a device for identifying fraud in store product display images, including:

[0021] An image acquisition module, configured to acquire the location information of multiple stores and the corresponding product display image of each store;

[0022] An image recognition module, configured to group multiple stores according to the location information of the stores. For each group, perform the following steps:

[0023] Extract features from the product display images of each store in the group to obtain the first image feature;

[0024] Use a preset first autoencoder model to process the SKU information of the product display image to obtain the second image feature;

[0025] Fuse the first image feature and the second image feature to obtain a fusion feature matrix;

[0026] Calculate the similarity between pairwise feature vectors of the fusion feature matrix, and based on the calculated similarity result, identify the stores where there is fraud in the product display images.

[0027] Optionally, the process of identifying the stores where there is fraud in the product display images based on the calculated similarity result is specifically as follows:

[0028] Determine whether the calculated similarity is greater than a preset threshold;

[0029] If there is a similarity greater than the preset threshold, based on the index of the fusion feature matrix, determine the corresponding store where there is falsehood in the product display image.

[0030] Optionally, the fusion processing of the first image feature and the second image feature to obtain a fusion feature matrix is specifically as follows:

[0031] Horizontally splice the first image feature and the second image feature to obtain a spliced feature matrix;

[0032] Use a preset second autoencoder model to process the spliced feature matrix to obtain a fusion feature matrix.

[0033] In a third aspect, an embodiment of the present invention further provides a computing device, where the computing device includes:

[0034] A communication interface for communicating with other devices;

[0035] A processor coupled to the communication interface, so that the communication device executes the method for identifying falsehood in the product display image of the store described in the first aspect.

[0036] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program runs on a computer, the computer is enabled to execute the method for identifying falsehood in the product display image of the store described in the first aspect.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] The method for identifying falsehood in the product display image of the store provided by the present invention first groups a large number of stores based on the store geographical location information. For the product display images obtained by business representatives visiting and taking pictures of each store within the group, two heterogeneous features, namely the SKU recognition content and the image feature of the image, are respectively obtained to fuse the two, so as to calculate the similarity based on the fusion feature, and then obtain the falsehood recognition result. The SKU recognition result is the semantic annotation of the display. Therefore, after learning the apparent similarity through the image feature and then fusing and corroborating it with the learning of the semantic annotation, the accuracy of identifying falsehood in similar product display images can be effectively improved, and the recognition efficiency is higher than that of the manual sampling inspection method. Description of the Drawings

[0039] To more clearly illustrate the technical solution of the present invention, the accompanying drawings required for the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0040] Figure 1 is a schematic flowchart of a method for identifying fraud in store commodity display images provided by an embodiment of the present invention;

[0041] Figure 2 is a schematic structural diagram of a device for identifying fraud in store commodity display images provided by an embodiment of the present invention. Specific Embodiments

[0042] The technical solution of the present invention will be further described in more detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0043] In the description of the present invention, it should be understood that terms such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that unless otherwise clearly specified and limited, the terms "connected" and "coupled" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium.

[0044] For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in combination with specific circumstances. In addition, in the description of the present invention, unless otherwise stated, the meaning of "a plurality" is two or more.

[0045] Any process or method description in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the technical field to which the embodiments of the present invention belong.

[0046] As Figure 1 shown, an embodiment of the present invention provides a method for identifying fraud in store commodity display images, including the following steps:

[0047] S1: Obtain the location information of multiple stores and the corresponding product display images for each store.

[0048] For the fast-moving consumer goods industry, it is often necessary to ensure a certain degree of coverage of terminal stores in order to increase product sales. Therefore, each fast-moving consumer goods company usually needs to manage multiple terminal stores simultaneously. During the operation visits to each terminal store, the sales representatives usually need to visit the store in person and take pictures of the product display in the store to obtain the product display images of each store, so that the company can calculate the activity costs based on the collected images.

[0049] Therefore, in the first step of this embodiment, first obtain the location information of multiple stores, specifically including the name and longitude and latitude information of the stores, etc.; then use the Geohash algorithm to encode and group the stores based on the obtained longitude and latitude information of the stores to avoid comparing all store data.

[0050] S2: Group the multiple stores according to the location information of the stores. For each group, perform the following steps:

[0051] S21: Extract features from the product display images of each store in the group to obtain the first image feature.

[0052] It can be understood that the sales representatives usually choose nearby stores as the collection source for fabricating product display images. Therefore, in this embodiment, the multiple stores are encoded and grouped based on the location information of each store to reduce invalid comparison operations, thereby improving the fraud recognition efficiency.

[0053] Specifically, for several stores in each group, respectively obtain the product display images corresponding to each store in the group, and extract features from the obtained images to obtain the first image feature.

[0054] In one embodiment, ResNet101 can be used as the backbone network to extract image features from the product display images of each store in the group to obtain the first image feature. The first image feature can specifically be a 2048-dimensional feature vector.

[0055] S22: Use a preset first autoencoder model to process the SKU information of the product display image to obtain the second image feature.

[0056] Specifically, for the product display images of each store in the group obtained, the SKU recognition result of the product display image can be obtained through an image recognition method.

[0057] It is understandable that after SKU recognition is performed on each product display image and expanded according to the categories and quantities of SKUs, the SKU recognition result of each product display image is flattened into an N-dimensional sparse feature after flattening.

[0058] In one embodiment, for the construction of a preset first autoencoder model, a sparse matrix of 50,000 SKU recognition results can be randomly selected as a training sample set to train the initial autoencoder model. Specifically, the preset first autoencoder model can be a variational autoencoder (VAE).

[0059] After processing the SKU information of the target product display image using the preset first autoencoder model, the obtained second image feature is specifically a low-dimensional dense representation feature vector of the SKU recognition result sparse matrix.

[0060] S23: Perform a fusion process on the first image feature and the second image feature to obtain a fusion feature matrix.

[0061] In one embodiment, the first image feature and the second image feature can be first concatenated horizontally to obtain a concatenated feature matrix, and then the preset second autoencoder model is used to process the concatenated feature matrix to obtain a fusion feature matrix. The fusion feature matrix contains feature vectors with a deeper fusion of two heterogeneous features.

[0062] Specifically, the preset second autoencoder model is a pre-trained autoencoder (AE).

[0063] S24: Calculate the similarity between pairwise feature vectors of the fusion feature matrix, and based on the calculated similarity results, identify the stores with fake product display images.

[0064] After calculating the similarity between pairwise feature vectors of the fusion feature matrix, first determine whether the obtained similarity is greater than a preset threshold; if there is a similarity greater than the preset threshold, it means that the store corresponding to this fusion feature matrix is a store with a fake product display image, that is, this store is a terminal store suspected of having fraudulent display fee investment behavior that needs to be tracked; therefore, based on the index of the fusion feature matrix, the corresponding store with a fake product display image can be determined.

[0065] Specifically, first, according to the calculated similarity results, group and save all feature vectors with feature similarities greater than the preset threshold. The grouping method is: divide the product display images belonging to the same GeoHash code into a repeated group, and then identify the stores with fake product display images.

[0066] It should be noted that each product display image is set with a unique identification code "id" to identify different product display images. At the same time, each product display image is also set with a store code "id" to identify product display images belonging to different stores. Each element in the feature matrix represents the feature value extracted from a product display image, and its subscript corresponds to the unique identification code "id" of the image. Therefore, the similar feature subscripts in the feature matrix can be traced based on the similarity, and then the corresponding product display image and its affiliated store can be traced.

[0067] In one embodiment, the similarity between the feature vectors of the fused feature matrix can be calculated through the Faiss framework. Correspondingly, the preset threshold can be set to 0.95, that is, when the similarity is greater than 0.95, it is determined that there may be a behavior of fabricating product display images in the store corresponding to the feature matrix.

[0068] It can be understood that the preset threshold can be determined according to the data set and the nature of the features, and the present invention does not make any limitations.

[0069] In one embodiment, based on the recognition result of the store with fabricated product display images obtained in the above embodiment, a data service can be released to provide data of the suspicious stores for relevant mini-program applications, so that the verification personnel can conduct offline verification on the suspicious stores. Specifically, during the verification process, the verification personnel can submit verification evidence by taking pictures.

[0070] Through the above embodiments of the present invention, a method for identifying fabrications of store product display images based on heterogeneous feature fusion is provided, which can quickly screen out terminal stores with fabrications of store product display images from a large amount of collected image data, achieving the goal of cost reduction and efficiency improvement.

[0071] As Figure 2 shown, another embodiment of the present invention further provides a device for identifying fabrications of store product display images, including an image acquisition module 101 and an image recognition module 102.

[0072] The image acquisition module 101 is used to acquire the location information of multiple stores and the product display images corresponding to each store.

[0073] The image recognition module 102 is used to group the multiple stores according to the location information of the stores. For each group, the following steps are executed:

[0074] Extract features from the product display images of each store in the group to obtain the first image feature; use a preset first autoencoder model to process the SKU information of the product display images to obtain the second image feature; perform fusion processing on the first image feature and the second image feature to obtain a fusion feature matrix; calculate the similarity between pairwise feature vectors of the fusion feature matrix, and identify the store with fake product display images according to the calculated similarity results.

[0075] Furthermore, the information interaction, execution process, etc. among the above-mentioned store product display image forgery recognition devices are based on the same concept as the embodiment of the store product display image forgery recognition method provided by the present invention. For specific content, reference can be made to the description in the method embodiment of the present invention, which will not be elaborated here.

[0076] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the method of this embodiment.

[0077] In a third aspect, the present invention provides a data processing device, including a processor, the processor is coupled to a memory, the memory stores a program, and the program is executed by the processor, so that the data processing device executes the store product display image forgery recognition method described in the first aspect.

[0078] In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the store product display image forgery recognition method described in the first aspect above.

[0079] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0080] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "serving as an example, instance, or illustration" and does not mean "preferred" or "advantageous" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, the technology can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0081] The foregoing description of the present disclosure has been provided to enable any ordinary person skilled in the art to make or use the present disclosure. Various modifications to the present disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying forged store product display images, characterized in that, Including: Obtain the location information of multiple stores and the corresponding product display images for each store; Group the multiple stores according to the location information of the stores. For each group, perform the following steps: Use ResNet101 as the backbone network to extract image features from the product display images of each store in the group to obtain the first image features; Use a preset first autoencoder model to process the SKU information of the product display images to obtain second image features, where the preset first autoencoder model is specifically: a variational autoencoder model, and the second image features are low-dimensional dense representation feature vectors of the sparse matrix of SKU recognition results; Perform fusion processing on the first image features and the second image features to obtain a fusion feature matrix; Calculate the similarity between the pairwise feature vectors of the fusion feature matrix, and identify the stores with fake product display images according to the calculated similarity results; Among them, the performing fusion processing on the first image features and the second image features to obtain a fusion feature matrix is specifically: horizontally splice the first image features and the second image features to obtain a spliced feature matrix; use a preset second autoencoder model to process the spliced feature matrix to obtain a fusion feature matrix, where the second autoencoder model is a pre-trained autoencoder.

2. The method for identifying false store merchandise display images according to claim 1, wherein The identifying the stores with fake product display images according to the calculated similarity results is specifically: Judge whether the calculated similarity is greater than a preset threshold; If there is a similarity greater than the preset threshold, based on the index of the fusion feature matrix, determine the corresponding store with a fake product display image.

3. An apparatus for identifying image forgery of store merchandise display, characterized in that, Including: An image acquisition module for obtaining the location information of multiple stores and the corresponding product display images for each store; An image recognition module for grouping the multiple stores according to the location information of the stores. For each group, perform the following steps: Use ResNet101 as the backbone network to extract image features from the product display images of each store in the group to obtain the first image features; Use a preset first autoencoder model to process the SKU information of the product display images to obtain second image features, where the preset first autoencoder model is specifically: a variational autoencoder model, and the second image features are low-dimensional dense representation feature vectors of the sparse matrix of SKU recognition results; Perform fusion processing on the first image features and the second image features to obtain a fusion feature matrix; Calculate the similarity between the pairwise feature vectors of the fusion feature matrix, and identify the stores with fake product display images according to the calculated similarity results; Among them, the performing fusion processing on the first image features and the second image features to obtain a fusion feature matrix is specifically: horizontally splice the first image features and the second image features to obtain a spliced feature matrix; use a preset second autoencoder model to process the spliced feature matrix to obtain a fusion feature matrix, where the second autoencoder model is a pre-trained autoencoder.

4. The store product display image forgery recognition device according to claim 3, wherein, Based on the calculated similarity results, identify the stores with false commodity display images, specifically as follows: Determine whether the calculated similarity is greater than a preset threshold; If there is a similarity greater than the preset threshold, based on the index of the fusion feature matrix, determine the corresponding store with false commodity display images.

5. A computing device, characterized in that, The computing device includes: A communication interface for communicating with other devices; A processor coupled to the communication interface, enabling the communication device to execute the method for identifying false commodity display images in stores according to any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which when run on a computer, causes the computer to execute the method for identifying false commodity display images in stores according to any one of claims 1 to 2.

Images