An automatic management system for large-screen electronic labels in the fresh food area

Abstract

The present invention discloses a large-screen electronic label automatic management system for a fresh food area, comprising a base station, a first shooting device, a regional control device and a remote server; the regional control device comprises a price change data table, a price adjustment module, a trend analysis module and a freshness evaluation model for multiple fresh goods; the price change data table is used to store the type of fresh goods, the freshness level of the goods and the corresponding price value; the freshness evaluation model evaluates the overall freshness level of the specified fresh goods and feeds the evaluation result back to the trend analysis module; the trend analysis module calculates the freshness change trend and generates a replenishment instruction or a price adjustment instruction according to the freshness change trend. The present invention greatly reduces the time for manual adjustment and confirmation of product information, and the verification result has high reliability, so that the product type observable by the user is consistent with the price tag display, thereby improving the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic labels for fresh food areas, and in particular to an automatic management system for large-screen electronic labels for fresh food areas. Background Art

[0002] Electronic paper electronic tags have become popular in supermarkets due to their excellent display effects and convenient price updates, and are gradually replacing the original paper tags. When the price of a product needs to be adjusted, you only need to send the adjusted product price image to the corresponding electronic tag unit to achieve price updates throughout the supermarket. Electronic tags come in a variety of sizes. Taking supermarkets as an example, 4-6-inch electronic tags may be used on shelves in department stores with a wide variety of products, while large-screen electronic tags of more than 10 inches will be used in the fresh food area, and customers will be provided with price reminder services by hanging them above fresh products.

[0003] At present, most electronic tags are updated through manual background processing, and a small number of research schemes have proposed to automatically update electronic tags after identifying the goods through some reading devices. The invention with application number 201810299344.5 published on October 18, 2019 discloses a method for automatic identification of commodity information, which can automatically adjust the commodity information on the electronic shelf label in combination with the technology of automatic identification of commodities, greatly improving work efficiency, while avoiding inconsistent information caused by the wrong delivery of commodities. However, this application uses RFID readers to realize the identification of commodity information, and can only update the electronic tags corresponding to a single commodity. Once the electronic tag corresponds to multiple commodities and the location area occupied by the commodity changes, the automatic update of the price tag cannot be realized. At present, there is no method for automatically updating the price tags of large-screen electronic tags that display multiple commodities at the same time.

[0004] In addition, the shelf life of fresh food products is short, and many fresh food products will lose moisture and have a lower chance of being sold within the same day. Usually, merchants will use price cuts and other promotions to increase the chance of selling. Currently, in order to address this phenomenon, staff members check the status of fresh food products and manually change them in the background, and then display the change results on the electronic tags. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides an automatic management system for large-screen electronic labels in the fresh food area, which can estimate the freshness level of the goods in the fresh food area and provide solutions for replenishment or price reduction promotions based on the estimation results. It can automatically adjust the price tags of goods according to the promotion plan pre-set by the user, greatly reducing the time for manual adjustment and confirmation of product information.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] An automatic management system for large-screen electronic tags in a fresh food area, used to manage and update the display content of electronic tags installed in the fresh food area; the automatic management system includes a base station, a first shooting device, a regional control device and a remote server;

[0008] The remote server generates a price tag for each fresh food product locally and sends it to a corresponding electronic tag through a base station; the electronic tag uses an electronic paper display screen to display the price tag of at least one fresh food product on the shelf;

[0009] The first photographing device is installed below the electronic tag and connected to the area control device. The first photographing device photographs the image of the fresh product in the designated product placement area corresponding to the electronic tag and sends the photographing result to the area control device.

[0010] The regional control device includes a price change data table, a price adjustment module, a trend analysis module and a plurality of fresh commodity freshness evaluation models; the price change data table is used to store the fresh commodity type, commodity freshness level and corresponding price value;

[0011] The freshness evaluation model is trained by using a plurality of stacked sample images of similar fresh products. The freshness evaluation model receives the fresh product images of the designated product placement area taken by the first shooting device, evaluates the overall freshness level of the designated fresh product, and feeds the evaluation results back to the trend analysis module.

[0012] The trend analysis module analyzes the overall freshness grade sequence data of each designated fresh commodity within a certain period of time, calculates its freshness change trend, and generates a replenishment instruction or a price adjustment instruction according to the freshness change trend; the trend analysis module is constructed by using a neural network, and is obtained by training a fitting curve of the overall freshness grade of fresh commodities over time under different external factors; the input of the trend analysis module is the historical freshness grade data of the fresh commodities corresponding to the m sampling periods counted from the current moment forward, and the output is the estimated freshness grade data of the m sampling periods counted from the current moment backward;

[0013] The price adjustment module updates the price tag of the corresponding commodity in combination with the price adjustment instruction and the price change data table.

[0014] To optimize the above technical solutions, the specific measures taken also include:

[0015] Further, the freshness evaluation model includes a feature extraction unit, a first feature fusion unit, a second feature fusion unit, an adaptive maximum pooling layer, a global average pooling layer and a fully connected layer connected in sequence;

[0016] The feature extraction unit includes three convolution modules, which perform layer-by-layer feature extraction on the imported fresh product images of the designated product placement area; the first feature fusion unit is used to extract the outputs of the second convolution module and the third convolution module, and perform feature fusion of different sizes; the second feature fusion unit is used to perform feature fusion again on the output of the first feature fusion unit and the output of the first convolution module; the features fused by the second feature fusion unit are subjected to adaptive maximum pooling and global average pooling processing by the adaptive maximum pooling layer and the global average pooling layer respectively, and the global features of the fresh product images are extracted; the fully connected layer is used to classify the extracted global features of the fresh product images, calculate the probability that the global features of the fresh product images belong to each freshness level, and output the freshness level of the current fresh product image.

[0017] Furthermore, the process of collecting training samples used by the freshness evaluation model includes:

[0018] For designated fresh products of different quantities and freshness levels, stir them at intervals and collect initial sample images in batches during the stirring intervals;

[0019] The initial sample images of each batch are enhanced by scaling, rotating, transforming, and flipping to expand the dataset.

[0020] Furthermore, the remote server includes an electronic tag management module, a commodity management module, a base station management module and a price update module; the tag management module is used to manage the electronic tags; the base station management module is used to manage the base stations; the commodity management module is used to manage the commodity types, supply quantities and sales quantities in the fresh food area; the price update module is used to receive or generate price tags for all commodities, and send the price tags to the electronic tags through the base stations for display; when one electronic tag corresponds to multiple commodities, the price update module is also used to splice the price tags of multiple commodities, and send the spliced ​​label images to the electronic tags for display.

[0021] Furthermore, the automatic management system further comprises a second shooting device distributed in the fresh food area;

[0022] The second photographing device is used to photograph the shelf image of the unit product area corresponding to each electronic tag within a preset field of view centered on the unit product area corresponding to the electronic tag;

[0023] The price update module includes a product identification unit, a placement area analysis unit and a label image stitching unit;

[0024] The product identification unit preliminarily divides the shelf image based on the unit product area, performs similarity judgment on the images of adjacent unit product areas, regards the adjacent unit product areas with similarity higher than the similarity threshold as the same product placement area, preliminarily integrates the unit product areas on the shelf image, generates a plurality of product placement areas, and identifies each product placement area; then calls the product identification model to identify the product type of each product placement area;

[0025] The placement area analysis unit defines the product placement area corresponding to only one electronic tag as the first placement area according to the frequency of occurrence of the product placement area, defines the remaining product placement areas as the second placement areas, and determines the main tag of each second placement area according to the image area ratio;

[0026] The label image splicing unit retrieves the product label image corresponding to each product placement area, splices the retrieved product label images according to the positional relationship between the various product placement areas on the shelf image, and generates an electronic label price tag; wherein the label image of the second placement area other than the main label is scaled according to the area occupied by the second placement area on the shelf image.

[0027] Furthermore, when one of the product placement areas contains more than two products, the product types it contains are matched with the product types of the adjacent product placement areas. If all product types are matched successfully, the product placement area is cleared; otherwise, a corresponding product placement area is generated according to the product types that failed to match; adjacent product placement areas refer to product placement areas that are within the coverage of the same electronic tag display.

[0028] Furthermore, the label image stitching unit includes a product label retrieval component, a template generation component, a judgment component, a scaling component and a filling component;

[0029] The product label retrieval component reads the product placement area number and the product type on each product placement area contained in the current collection cycle for each electronic label, and retrieves the product label image data corresponding to each product placement area according to the product type;

[0030] The template generation component generates a label module according to the position information of the unit product area occupied by each product placement area, and the label template includes a plurality of label filling boxes, each label filling box corresponds to a product placement area;

[0031] The judging component is used to judge whether the electronic label corresponds to a second placement area other than the main label, and if so, call the scaling component to scale the corresponding label filling frame within a preset image size range according to the area occupied by the second placement area on the shelf image;

[0032] The filling component fills the retrieved commodity label image data into the corresponding label filling frame to generate an electronic label price tag within the current collection cycle.

[0033] Further, the verification module includes a label image acquisition unit, a verification information retrieval unit, a first judgment unit, a scaling recalculation unit, a second judgment unit, a third judgment unit and a judgment result output unit;

[0034] The label image acquisition unit combines the electronic label and the product placement area relationship data table to acquire a display image data set of the electronic label in the same second placement area a.

[0035] The verification information retrieval unit calls the original collected shelf image data set corresponding to the aforementioned electronic tag According to the position information of the unit product area occupied by the second placement area a, the display image data set of the electronic tag is The corresponding image data set of the second placement area a is obtained by extraction {a (m)}; m = 1, 2, ..., M;

[0036] The first judging unit judges the image data set {a (m)} whether the product label image data in is complete, if complete, send a start signal to the scaling recalculation unit, otherwise, call the judgment result output unit to output the update failure result;

[0037] The scaling ratio recalculation unit calculates the second placement area a in the shelf image data set The scale ratio set of the label filling box of the second display area a is calculated by calculating the area occupied by each shelf image in

[0038] The second judgment unit sets the scaling ratio according to the scaling ratio. For the data image data set {a (m)} to obtain the scaled validation set {b (m)}, Then judge the validation set {b (m)}, if they are consistent, sending a start signal to the third judgment unit, otherwise, calling the judgment result output unit to output the update failure result;

[0039] The third judgment unit performs a check on the verification set {b (m)} and the similarity between the price tag of the corresponding product on the second display area a. If the similarity is higher than the preset similarity threshold, the judgment result output unit is called to output the update success result; otherwise, the judgment result output unit is called to output the update failure result.

[0040] The beneficial effects of the present invention are:

[0041] First, the large-screen electronic label automatic management system for the fresh food area of ​​the present invention can estimate the freshness level of the goods in the fresh food area, and provide solutions for replenishment or price reduction promotions based on the estimation results. It can automatically adjust the price tags of goods according to the promotion plan pre-set by the user, greatly reducing the time for manual adjustment and confirmation of product information.

[0042] Second, the large-screen electronic label automatic management system of the fresh food area of ​​the present invention can automatically adjust the display image of the large-screen electronic label according to the real-time status of the goods on the shelf; based on the user's visual range, the user can obtain the price information of the product type (including misplaced goods) within the close visual range at any angle, thereby improving the user experience. In addition, since the price label display content can include the price information of all misplaced goods, the probability of manual rearrangement can be reduced by attracting users to buy.

[0043] Thirdly, the large-screen electronic label automatic management system for the fresh food area of ​​the present invention can realize automatic verification of the update results in combination with the repeated price labels of some commodities, and the verification results are highly reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a structural diagram of the large-screen electronic label automatic management system for the fresh food area of ​​the present invention.

[0045] Figure 2 It is a structural diagram of the freshness evaluation model of the present invention.

[0046] Figure 3 It is a structural diagram of the price update module of the present invention.

[0047] Figure 4 It is a structural diagram of the verification module of the present invention. DETAILED DESCRIPTION

[0048] The present invention will now be described in further detail with reference to the accompanying drawings.

[0049] It should be noted that the terms such as "upper", "lower", "left", "right", "front", "back", etc. cited in the invention are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments in their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0050] Figure 1 This is the structure diagram of the large-screen electronic label automatic management system for the fresh food area of ​​the present invention. Figure 1The automatic management system is used to manage and update the display content of the electronic tags installed in the fresh food area. The automatic management system includes a base station, a first shooting device, a regional control device and a remote server.

[0051] The remote server generates a price tag for each fresh food product locally and sends it to the corresponding electronic tag through the base station; the electronic tag uses an electronic paper display screen to display the price tag of at least one fresh food product on the shelf.

[0052] The first photographing device is installed below the electronic tag and connected to the area control device. The first photographing device photographs the image of the fresh goods in the designated product placement area corresponding to the electronic tag, and sends the photographing result to the area control device.

[0053] The regional control device includes a price change data table, a price adjustment module, a trend analysis module and a freshness evaluation model for multiple fresh products; the price change data table is used to store the fresh product type, product freshness level and corresponding price value.

[0054] The freshness evaluation model is trained using a number of stacked sample images of similar fresh products. The freshness evaluation model receives images of fresh products in a designated product placement area taken by a first camera, evaluates the overall freshness level of the designated fresh products, and feeds the evaluation results back to the trend analysis module.

[0055] The trend analysis module analyzes the overall freshness grade sequence data of each designated fresh product within a certain time period, calculates its freshness change trend, and generates replenishment instructions or price adjustment instructions based on the freshness change trend; the trend analysis module is trained using the overall freshness grade sequence data and sales sequence data based on unit time.

[0056] The price adjustment module updates the price tags of the corresponding products in combination with the price adjustment instructions and the price change data table.

[0057] 1. Freshness Judgment

[0058] See also Figure 2 ,The freshness evaluation model includes a feature extraction unit, a first feature fusion unit, a second feature fusion unit, an ,adaptive maximum pooling layer, a global average pooling layer and a ,fully connected layer in sequence.

[0059] The feature extraction unit includes three convolution modules, which perform layer-by-layer feature extraction on the imported fresh product images of the designated product placement area; the first feature fusion unit is used to extract the outputs of the second convolution module and the third convolution module, and perform feature fusion of different sizes; the second feature fusion unit is used to perform feature fusion again on the output of the first feature fusion unit and the output of the first convolution module; the features fused by the second feature fusion unit are subjected to adaptive maximum pooling and global average pooling processing by the adaptive maximum pooling layer and the global average pooling layer respectively, and the global features of the fresh product images are extracted; the fully connected layer is used to classify the extracted global features of the fresh product images, calculate the probability that the global features of the fresh product images belong to each freshness level, and output the freshness level of the current fresh product image.

[0060] The first feature fusion unit is used to fuse the middle-level feature values ​​output by the second convolution module and the deep-level feature values ​​output by the third convolution module to obtain the first fused feature, which is used to characterize the semantic information of fresh products. The second feature fusion unit is used to fuse the middle-level feature values ​​output by the second convolution module and the deep-level feature values ​​output by the third convolution module to obtain the first fused feature. The first convolution module focuses more on extracting the geometric detail features of fresh products. The second feature fusion unit is then used to perform feature fusion again on the first fused feature (which focuses more on the epidermal semantic features of fresh products) and the shallow feature values ​​output by the first convolution module (which focuses more on the geometric detail features of fresh products) to obtain the global features of the fresh product image. In addition, this embodiment judges the overall freshness of fresh products, while taking into account the accuracy and efficiency of the evaluation results, and only needs to use a three-layer convolutional network to execute, realizing the lightweight computing requirements of the fresh area.

[0061] The process of collecting training samples used by the freshness evaluation model includes:

[0062] For designated fresh products of different quantities and different freshness levels, stir them at intervals, and collect initial sample images in batches during the stirring intervals; enhance the initial sample images of each batch, and use multiple methods such as scaling, rotation, transformation, and flipping to expand the data set. For example, the freshness level of a certain fresh product is divided into five levels: 100%, 80%, 60%, 40%, and 20%. For the five freshness levels, four quantities of 200, 100, 50, and 20 are set respectively, totaling 20 kinds of original fresh product samples. Stir these 20 kinds of original fresh product samples, and collect images during the stirring intervals. This cycle is repeated to collect the initial sample image data. After enhancing the initial sample image data, it is expanded by scaling, rotation, transformation, and flipping, and finally 16,400 fresh image sample data of different resolutions can be obtained. In practical applications, the freshness level and quantity level can be set by yourself, and are not limited to the aforementioned one.

[0063] Considering the judgment efficiency, this embodiment proposes to judge the freshness of fresh products as a whole, rather than a single fresh product. In supermarkets, environmental factors such as lighting are usually consistent. Even if the ambient light of some products is reduced due to human body occlusion and other reasons, the influence of light intensity on freshness judgment can be ignored by just filtering the collected images accordingly. On this basis, the change in freshness of fresh products is related to two factors: the amount of water loss that changes over time, and the change in the freshness ratio of the remaining products caused by customer selection. For the former, the freshness change curve caused by water loss is relatively flat. For the latter, customers are more inclined to choose fresher products. Therefore, the behavior of customers buying products will cause the freshness of the remaining products to change suddenly. For example, for a fresh product, such as apples, since supermarket warehouses usually have a fresh-keeping area, we assume that the apples just placed in the product display area are fresh (from an overall perspective, not for individual apples). As time goes by, the moisture of individual apples will be lost, and since customers are more inclined to choose fruits with higher freshness, the overall freshness level of apples in the product display area continues to decline.

[0064] On the basis of obtaining the real-time freshness of fresh goods, it is certainly possible to perform price tag adjustments or generate replenishment requests based on their real-time freshness, but more preferably, this embodiment combines historical freshness level data to estimate the freshness of fresh goods, and performs subsequent actions based on the estimated results. This embodiment uses a GRU network model based on an attention mechanism to achieve the aforementioned technical effects. The input of the trend analysis module is the historical freshness level data of fresh goods corresponding to the m sampling periods counted from the current moment forward, and the output is the estimated freshness level data of the m sampling periods counted from the current moment backward. Preferably, the trend analysis module is trained using the freshness level sequence data of fresh goods within a certain time period and the freshness change sample data caused by different sales volumes under each freshness level condition.

[0065] The trend analysis model of this embodiment is based on the following shopping assumptions obtained from historical data analysis: the sales volume under the same freshness level tends to be stable, but is related to external factors such as season, price level and the number of similar products currently provided by the supermarket. When the overall freshness level of fresh products is very high, it will stimulate the user's shopping desire, the sales volume per unit time is high, and the overall freshness level change curve over time contains more frequent mutations; with the passage of time and the sale of goods, the overall freshness level of fresh products gradually decreases, the user's shopping desire is also getting lower and lower, the sales volume per unit time will gradually decrease, and the overall freshness level change curve over time contains fewer mutations, and the curve will be flatter. Therefore, when the external factors are determined, the change curve of the overall freshness level of fresh products is relatively fixed. This embodiment collects the historical freshness level data of fresh products corresponding to m sampling periods, and at the same time estimates the estimated freshness level data of m sampling periods counted from the current moment backward.

[0066] Exemplarily, the trend analysis module can be constructed using a neural network, which is obtained by training a fitting curve of the overall freshness level of fresh products changing over time under different external factors.

[0067] Still taking apples as an example, the training process of the trend analysis module is as follows: under different external factors, the overall freshness level sequence data of apples within a certain time range is collected, with time as the horizontal axis and the overall freshness level as the vertical axis, and the overall freshness level curve of apples within the time range is fitted as a set of sample data. The trend analysis module is trained using the sample data set, so that the trend analysis module can estimate the freshness change curve of apples in the next m sampling periods based on the historical freshness level data of apples in the imported m sampling periods, and perform price tag adjustment or generate replenishment requests based on the estimated results. For example, when the freshness level of a product drops to 40% freshness level, the discount price corresponding to the freshness level 40% in the price change data table is consulted, and the price of the electronic tag of the product is automatically updated; or when it is estimated that the freshness level of the product will drop rapidly to 40% freshness level in the short term, the product is replenished to avoid sales gaps and miss the golden sales time.

[0068] Preferably, this embodiment also mentions a method for managing commodity prices with maximizing sales as the optimization goal. Specifically, the overall freshness level and sales volume change curve of fresh commodities under different external factors are obtained in combination with historical data. By analyzing the estimated freshness level change curve, the estimated sales volume change curve of the commodity is determined, and the corresponding price adjustment strategy is automatically executed according to the estimated result. For example, when the freshness level of the commodity drops to 40% of the freshness level and the freshness will continue to decline slowly during the estimated time period, it means that the sales volume is difficult to increase in the short term. At this time, the discount price corresponding to the freshness level of 40% in the price change data table is consulted, and the price of the electronic tag of the commodity is automatically updated. On the contrary, if the freshness level of the commodity drops to 40% of the freshness level but the freshness level will still decrease significantly during the estimated time period, it means that even if the freshness is low, the estimated sales volume of the commodity is still large, and the customer's psychological acceptance of the commodity is still high. At this time, you can choose to continue to wait and see. This embodiment can reduce manual intervention, estimate the sales volume of goods by estimating the freshness level, manage the price of goods with the optimization goal of maximizing sales, and reduce the workload of manual inspection of goods and manual update of price tags (even updating electronic tags). The price change data table can be maintained and updated by the remote server.

[0069] 2. Display of multiple product price tags

[0070] Due to the wide variety of goods in the fresh food area, a medium-height platform structure is usually used, and a dozen or even dozens of unit product areas are set up on one platform. The merchant sets a 10-inch electronic tag for each product and places it in front of the corresponding product to indicate the price. The user needs to match the product with the electronic tag one by one and then move to view the product price. To this end, this embodiment proposes to display multiple product prices on one electronic tag at the same time.

[0071] In this embodiment, the display content of the electronic tag is affected by the user's visual range, that is, the electronic tag in each fresh food area should be able to display the prices of other commodities within a certain field of vision of the user centered on the corresponding commodity. Compared with setting a number of electronic tags corresponding to the number of commodities according to the placement of commodities, this greatly reduces the effort required by users to find prices.

[0072] The collection of shelf images can be achieved through a second shooting device distributed in the fresh food area, or it can be combined with manual patrol shooting. If it is necessary to combine manual patrol shooting, the images taken by manual patrol can be calibrated again, or the correlation between them and the images taken at fixed points can be indirectly calculated, and then used for price update applications. In this embodiment, in addition to being executed when the goods are put on the shelves, price updates can also be reflected in the daily operation and maintenance of the supermarket. For example, if one of the goods on a shelf is sold out, the price tag can be adjusted in time through regular updates to avoid misunderstandings by users.

[0073] See also Figure 3 ,The price update module includes a product recognition unit, a placement area analysis unit and a label image stitching unit.

[0074] The product recognition unit preliminarily divides the shelf image based on the unit product area, makes a similarity judgment on the images of adjacent unit product areas, regards the adjacent unit product areas with a similarity higher than a similarity threshold as the same product placement area, preliminarily integrates the unit product areas on the shelf image, generates several product placement areas, and identifies each product placement area; then, the product recognition model is called to identify the product type in each product placement area.

[0075] The process of the product identification unit identifying product information includes the following sub-steps:

[0076] Collect several product image samples contained in the supermarket to construct a training set. The product image samples include stacked image samples of multiple similar products. Construct a product recognition model based on a neural network, and use the training set to train the product recognition model. The product recognition model can adopt the image recognition technology in the prior art. For example, the invention with application number 202010874612.9 published on October 2, 2020 discloses a product recognition management method, which obtains the category and model information of the product by training and recognizing the image of the target product. Take the unit product area as the unit, preliminarily divide the shelf image, judge the similarity of the images of adjacent unit product areas, and regard the adjacent unit product areas with similarity higher than the similarity threshold as the same product placement area; preliminarily integrate the unit product areas on the shelf image, generate several product placement areas, and mark each product placement area. Import the image of each product placement area into the product recognition model, and identify the product type of each product placement area.

[0077] When one of the product placement areas contains more than two kinds of goods, the types of goods contained therein are matched with the types of goods in the adjacent product placement area. If all the types of goods are matched successfully, the product placement area is cleared, otherwise, the corresponding product placement area is generated according to the types of goods that are not matched successfully; the adjacent product placement area refers to the product placement area that belongs to the same electronic tag display coverage. This is because in a supermarket environment, due to the open shopping environment, it is often the case that users take the goods and put them in another area by mistake. In the prior art, for this situation, it is usually discovered through manual inspections or image recognition combined with database proofreading, and then the position of the goods is corrected by notifying the staff to re-place the goods. In this embodiment, it is not regarded as a simple misplaced product, but as a secondary product. Its price information is displayed simultaneously in the large-screen electronic tag, so as to increase the probability of users purchasing the product and reduce the workload of the staff to pick and place.

[0078] The placement area analysis unit defines the product placement area corresponding to only one electronic tag as the first placement area and the remaining product placement areas as the second placement areas according to the frequency of occurrence of the product placement areas, and determines the main label of each second placement area according to the image area ratio.

[0079] In this embodiment, in addition to the corresponding misplaced secondary selected commodities, the commodities on the second display area also include commodities in the same viewing direction that are less likely to attract the user's attention because of the small area occupied. These commodities occupy a small area in the viewing direction A due to the angle problem and can only be partially displayed, but they may be fully displayed in the viewing direction B (for example, an electronic tag is located directly in front of commodity 1. Therefore, when the shelf image is collected with the electronic tag orientation as the viewing center, commodity 1 will occupy a large area, making the electronic tag the main label of commodity 1). Of course, in some examples, the main label of the commodity can also be specified by binding the relationship between the label and the commodity in the background; but the confirmation process of the previous main label is more in line with the visual law of the user when shopping, especially when the commodity is displaced, the commodity is repeatedly placed in multiple unit product areas, or the label position is offset, the self-correction of the electronic label price display can still be achieved. Combined with the characteristics of multi-commodity display, on the basis of ensuring that the commodity price is fully displayed, it is ensured that customers can continuously and quickly obtain correct price information.

[0080] The label image splicing unit retrieves the product label image corresponding to each product placement area, splices the retrieved product label images according to the positional relationship of each product placement area on the shelf image, and generates an electronic label price tag; wherein, the label image of the second placement area other than the main label is scaled according to the area occupied by the second placement area on the shelf image.

[0081] The label image stitching unit includes a product label retrieval component, a template generation component, a judgment component, a scaling component and a filling component. The product label retrieval component reads the product placement area number and the product type on each product placement area contained in the current collection cycle for each electronic label, and retrieves the product label image data corresponding to each product placement area according to the product type; the template generation component generates a label module according to the position information of the unit product area occupied by each product placement area, and the label template contains several label filling boxes, each label filling box corresponds to a product placement area; the judgment component is used to determine whether the electronic label corresponds to the second placement area other than the main label. If so, the scaling component is called to scale the corresponding label filling box within the preset image size range according to the area occupied by the second placement area on the shelf image; the filling component fills the retrieved product label image data into the corresponding label filling box to generate an electronic label price tag within the current collection cycle.

[0082] After obtaining the correspondence between the electronic tag and the product placement area, the splicing process of the product price tag can be realized according to the positional relationship of the product placement area. The electronic tag price board of this embodiment is spliced ​​according to the positional relationship of the product, which can correctly guide customers to understand the product information; at the same time, the product label image in the electronic tag price board is related to the user's attention under the current viewing angle, which can reduce the effort required by the user to obtain price information and reduce the misjudgment rate.

[0083] In this embodiment, although the product label image is scaled, it is still within the reasonable range of human vision, and the size and number of the electronic labels corresponding to the shelf can be determined in combination with the type and quantity of the products on the shelf.

[0084] Finally, the label image stitching unit sends the stitched electronic label price tag to the corresponding electronic label for display.

[0085] In this embodiment, the purpose of setting the second display area is not only to enable the user to fully and quickly understand the prices of all commodities within the field of vision, but also has another important significance, that is, to serve as a verification standard for the update results.

[0086] At present, the verification of the price tag display results is still done manually. There are many types of goods in the fresh food area, and the probability of promotion and discount is also high. Even experienced staff find it difficult to remember the real-time prices of most goods. If the update is more frequent, the workload of the staff will increase exponentially. This is especially true for large-screen price tags that display prices of multiple goods. To this end, this embodiment proposes to perform consistency verification on the price tag image data of the second display area that is repeatedly displayed to achieve an estimation of the display results of the entire electronic tag.

[0087] See also Figure 4 , the remote server also includes a verification module; the verification module includes a label image acquisition unit, a verification information retrieval unit, a first judgment unit, a scaling recalculation unit, a second judgment unit, a third judgment unit and a judgment result output unit;

[0088] The label image acquisition unit combines the electronic label and the product placement area relationship data table to collect the display image data set of the electronic label in the same second placement area a. The collection method includes directly photographing the price tag image of the electronic tag, or requesting the electronic tag to feed back the price tag image data through a base station.

[0089] The verification information retrieval unit calls the original collected shelf image data set corresponding to the aforementioned electronic tag According to the position information of the unit product area occupied by the second placement area a, the display image data set of the electronic tag is The corresponding image data set of the second placement area a is extracted from (m)}; m = 1, 2, ..., M.

[0090] The first judging unit judges the image data set {a (m)} is complete. If complete, a start signal is sent to the scaling recalculation unit. Otherwise, the judgment result output unit is called to output the update failure result.

[0091] The scaling recalculation unit calculates the second placement area a in the shelf image data set The scale ratio set of the label filling box of the second display area a is calculated by calculating the area occupied by each shelf image in

[0092] The second judgment unit sets the For the data image data set {a (m)} to obtain the scaled validation set {b (m)}, Determine the validation set {b (m)} are consistent, if consistent, send a start signal to the third judgment unit, otherwise, call the judgment result output unit to output the update failure result.

[0093] The third judgment unit performs a test on the validation set {b (m)} and the similarity between the image in the second display area a and the product price tag of the corresponding product on the second display area a. If the similarity is higher than the preset similarity threshold, the judgment result output unit is called to output the update success result; otherwise, the judgment result output unit is called to output the update failure result.

[0094] Take the four-sided cargo platform as an example. Assume that there are several commodities 1 on the cargo platform on the east side. Some commodities 1 can also be observed from the south and north aisles. The four-sided cargo platform includes three electronic tags, one of which faces the east and corresponds to commodity 1, while the other two face the southwest and northwest, corresponding to commodities 2 and 3. After analysis, electronic tag 1 facing the east is the main tag, and electronic tags 2 and 3 facing the southwest and northwest are secondary tags. Electronic tags 1, 2, and 3 process and display the commodity price image of commodity 1 according to the area occupied and position of commodity 1 on its corresponding shelf image. In electronic tag 2, the price image of commodity 1 is smaller and displayed to the right, and in electronic tag 3, the price image of commodity 1 is smaller and displayed to the left.

[0095] During the verification process, the complete display images of electronic tags 1, 2, and 3 are collected respectively, and the corresponding display area price images are intercepted from the set positions in the complete display images according to the electronic tag template. If the display area price image is incomplete, it means that the current electronic tag update has failed, and it is very likely that the updated image data has not been received; otherwise, the original shelf image is processed again, the scaling ratio is calculated, and the intercepted display area price image is scaled and then compared for consistency, and the calculation results, scaling ratio, and image data display of the original shelf image are verified. The original shelf image here refers to the data basis when the electronic tag is updated, not just the image when the product is put on the shelf. This is for the verification work when the electronic tag is updated during the product sales process.

[0096] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. An automatic management system for large-screen electronic labels in the fresh food area, used to manage and update the display content of electronic labels installed in the fresh food area; characterized in that: The automatic management system includes a base station, a first shooting device, a regional control device and a remote server; The remote server generates a price tag for each fresh food product locally and sends it to a corresponding electronic tag through a base station; the electronic tag uses an electronic paper display screen to display the price tag of at least one fresh food product on the shelf; The first photographing device is installed below the electronic tag and connected to the area control device. The first photographing device photographs the image of the fresh product in the designated product placement area corresponding to the electronic tag and sends the photographing result to the area control device. The regional control device includes a price change data table, a price adjustment module, a trend analysis module and a plurality of fresh commodity freshness evaluation models; the price change data table is used to store the fresh commodity type, commodity freshness level and corresponding price value; The freshness evaluation model is trained by using a plurality of stacked sample images of similar fresh products. The freshness evaluation model receives the fresh product images of the designated product placement area taken by the first shooting device, evaluates the overall freshness level of the designated fresh product, and feeds the evaluation results back to the trend analysis module. The trend analysis module analyzes the overall freshness grade sequence data of each designated fresh commodity within a certain period of time, calculates its freshness change trend, and generates a replenishment instruction or a price adjustment instruction according to the freshness change trend; the trend analysis module is constructed using a neural network, and is obtained by training a fitting curve of the overall freshness grade of fresh commodities over time under different external factors; The input of the trend analysis module is the historical freshness level data of fresh products corresponding to the m sampling periods counted forward from the current moment, and the output is the estimated freshness level data of the m sampling periods counted backward from the current moment; The price adjustment module updates the price tag of the corresponding commodity in combination with the price adjustment instruction and the price change data table; The remote server includes an electronic tag management module, a commodity management module, a base station management module and a price update module; the tag management module is used to manage the electronic tags; the base station management module is used to manage the base stations; the commodity management module is used to manage the commodity types, supply quantities and sales quantities in the fresh food area; the price update module is used to receive or generate price tags for all commodities, and send the price tags to the electronic tags through the base stations for display; when one electronic tag corresponds to multiple commodities, the price update module is also used to splice the price tags of the multiple commodities, and send the spliced ​​label images to the electronic tags for display; The automatic management system further includes a second shooting device distributed in the fresh food area; The second photographing device is used to photograph the shelf image of the unit product area corresponding to each electronic tag within a preset field of view centered on the unit product area corresponding to the electronic tag; The price update module includes a product identification unit, a placement area analysis unit and a label image splicing unit; The product identification unit preliminarily divides the shelf image based on the unit product area, performs similarity judgment on the images of adjacent unit product areas, regards the adjacent unit product areas with similarity higher than the similarity threshold as the same product placement area, preliminarily integrates the unit product areas on the shelf image, generates a plurality of product placement areas, and identifies each product placement area; then calls the product identification model to identify the product type of each product placement area; The placement area analysis unit defines the product placement area corresponding to only one electronic tag as the first placement area according to the frequency of occurrence of the product placement area, defines the remaining product placement areas as the second placement areas, and determines the main tag of each second placement area according to the image area ratio; The label image splicing unit retrieves the product label image corresponding to each product placement area, splices the retrieved product label images according to the positional relationship of each product placement area on the shelf image, and generates an electronic label price tag; wherein the label image of the second placement area other than the main label is scaled according to the area occupied by the second placement area on the shelf image; The remote server also includes a verification module; the verification module includes a label image acquisition unit, a verification information retrieval unit, a first judgment unit, a scaling recalculation unit, a second judgment unit, a third judgment unit and a judgment result output unit; The label image acquisition unit combines the electronic label and the product placement area relationship data table to acquire a display image data set of the electronic label in the same second placement area a. The verification information retrieval unit calls the original collected shelf image data set corresponding to the aforementioned electronic tag According to the position information of the unit product area occupied by the second placement area a, the display image data set of the electronic tag is The corresponding image data set of the second placement area a is extracted from (m) }; m = 1, 2, ..., M; The first judging unit judges the image data set {a (m) } whether the product label image data in is complete, if complete, send a start signal to the scaling recalculation unit, otherwise, call the judgment result output unit to output the update failure result; The scaling ratio recalculation unit calculates the second placement area a in the shelf image data set The scale ratio set of the label filling box of the second display area a is calculated by calculating the area occupied by each shelf image in The second judgment unit sets the scaling ratio according to the scaling ratio. For the data image data set {a (m) } to obtain the scaled validation set {b (m) }, Then judge the validation set {b (m) }, if they are consistent, sending a start signal to the third judgment unit, otherwise, calling the judgment result output unit to output the update failure result; The third judgment unit performs a check on the verification set {b (m) } and the similarity between the price tag of the corresponding product on the second display area a. If the similarity is higher than the preset similarity threshold, the judgment result output unit is called to output the update success result; otherwise, the judgment result output unit is called to output the update failure result.

2. The large-screen electronic label automatic management system for the fresh food area according to claim 1 is characterized in that: The freshness evaluation model includes a feature extraction unit, a first feature fusion unit, a second feature fusion unit, an adaptive maximum pooling layer, a global average pooling layer and a fully connected layer connected in sequence; The feature extraction unit includes three convolution modules, which perform layer-by-layer feature extraction on the imported fresh product images of the designated product placement area; the first feature fusion unit is used to extract the outputs of the second convolution module and the third convolution module, and perform feature fusion of different sizes; the second feature fusion unit is used to perform feature fusion again on the output of the first feature fusion unit and the output of the first convolution module; The features fused by the second feature fusion unit are subjected to adaptive maximum pooling and global average pooling processing by the adaptive maximum pooling layer and the global average pooling layer respectively, so as to extract the global features of the fresh product image; the fully connected layer is used to classify the extracted global features of the fresh product image, calculate the probability that the global features of the fresh product image belong to each freshness level, and output the freshness level of the current fresh product image.

3. The large-screen electronic label automatic management system for the fresh food area according to claim 1 is characterized in that: The process of collecting training samples used by the freshness evaluation model includes: For designated fresh products of different quantities and freshness levels, stir them at intervals and collect initial sample images in batches during the stirring intervals; The initial sample images of each batch are enhanced by scaling, rotating, transforming, and flipping to expand the dataset.

4. The large-screen electronic label automatic management system for fresh food area according to claim 1 is characterized in that: When one of the product placement areas contains more than two kinds of goods, the types of goods it contains are matched with the types of goods in the adjacent product placement areas. If all product types are matched successfully, the product placement area is cleared. Otherwise, a corresponding product placement area is generated according to the product types that failed to match; adjacent product placement areas refer to product placement areas that are within the coverage of the same electronic tag display.

5. The large-screen electronic label automatic management system for the fresh food area according to claim 1 is characterized in that: The label image stitching unit includes a product label retrieval component, a template generation component, a judgment component, a scaling component and a filling component; The product label retrieval component reads the product placement area number and the product type on each product placement area contained in the current collection cycle for each electronic label, and retrieves the product label image data corresponding to each product placement area according to the product type; The template generation component generates a label module according to the position information of the unit product area occupied by each product placement area, and the label template includes a plurality of label filling boxes, each label filling box corresponds to a product placement area; The judging component is used to judge whether the electronic label corresponds to a second placement area other than the main label, and if so, call the scaling component to scale the corresponding label filling frame within a preset image size range according to the area occupied by the second placement area on the shelf image; The filling component fills the retrieved commodity label image data into the corresponding label filling frame to generate an electronic label price tag within the current collection cycle.

Images