Commodity label area image extraction method and device based on deep learning
Through the deep learning-based product label area segmentation method, the multi-scale feature fusion and spatial attention module are used to solve the problems of inaccurate and poor universality of the region of interest in the traditional method, and efficient extraction of different product label areas and Lighting robustness.
Patent Information
- Application Number
- CN202411522509.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2024-12-27
AI Technical Summary
The prior art has problems of inaccurate segmentation of areas of interest and waste of computing resources in product label area extraction, especially due to poor versatility caused by lighting inhomogeneity and differences in appearance of different products. Traditional algorithms require multiple prior parameters to be set, resulting in The non-labeled part is also divided into foreground.
Using the product label area segmentation method based on deep learning, a pre-trained product label area segmentation model is used. The model includes a downsampled feature extraction part, an upsampled feature fusion part and a pyramid pooling module, and fuses different through multi-scale ups and downsampling paths. hierarchical feature information improves the extraction effect of the region of interest, and enhances the robustness of the network to light changes through the spatial attention module.
It effectively improves the extraction accuracy and versatility of product label areas, can more accurately divide the price area, warranty period area and label defect area, reduces the impact on light changes, and is suitable for the extraction of label areas of different products.
Smart Images
Figure CN119206232A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the fields of smart retail and smart quality inspection technology, and in particular to a method and device for extracting product label area images based on deep learning. Background Art
[0002] For labels printed on products, people often pay more attention to the key areas, such as the price area, warranty period, defective area, etc. These areas usually occupy a small area of the entire label, and directly identifying or detecting related content on the entire image often misses important content; on the other hand, most of the entire image is non-interesting areas that do not need to be paid attention to, and detecting the entire image will also waste a lot of computing resources. In order to improve detection efficiency and accuracy, we need to first use an algorithm to segment the region of interest of the label. However, due to differences in appearance and shape of different products, the blank part without a label often has a highlight area. Traditional algorithms require too many prior parameter settings and have poor versatility, resulting in existing traditional algorithms also classifying these non-label parts as foregrounds.
[0003] For example, the binarization algorithm based on threshold segmentation: the traditional image binarization algorithm usually adopts threshold segmentation. For a certain pixel value, if it is lower than the preset threshold, it is divided into the background, and if it is higher than the threshold, it is the foreground. Then, the part of interest is selected according to some manually formulated rules. On the one hand, this algorithm relies on the setting of thresholds, and different thresholds are required for different scenes, which is not conducive to the automation of the entire process. On the other hand, since the illumination of each part of the entire image is not uniform, the result obtained directly through threshold segmentation is also very inaccurate. Although there are some improved adaptive binarization algorithms for these problems later, the effect is still unsatisfactory, and good results cannot be obtained for problems such as highlights and shadows. For different commodities, it is necessary to combine various different prior rules, resulting in the algorithm model often only applicable to specific commodities and lack of universality. Summary of the invention
[0004] The present invention provides a method and device for extracting product label area images based on deep learning.
[0005] According to a first aspect of the present disclosure, a method for extracting product label area images based on deep learning is provided. The method comprises:
[0006] Receiving product images;
[0007] The product image is input into a pre-trained product label region segmentation model to obtain a label segmentation result of the product image; the product label region segmentation model includes a downsampling feature extraction part, an upsampling feature fusion part and a pyramid pooling module, and the pyramid pooling module is configured with an intermediate convolution layer, so that the intermediate convolution layer upsamples the bottom feature map of the downsampling feature extraction part to obtain a target feature map with the same size as the feature map corresponding to each level of the upsampling feature fusion part, so that the product label region segmentation model splices the target feature map with the feature map corresponding to each level of the upsampling feature fusion part;
[0008] The product label area is extracted from the product image according to the label segmentation result to obtain a product label area image; the product label area includes at least one of a price area, a warranty period area and a label defect area.
[0009] According to the above aspects and any possible implementation, an implementation is further provided.
[0010] The product label region segmentation model also includes a convolutional neural network;
[0011] The pre-training process of the product label region segmentation model is completed through the following steps:
[0012] Obtain a binary image of the product label area of the multiple product images as the true value;
[0013] Input the multiple product images into a convolutional neural network for forward propagation, and after downsampling and upsampling, output feature maps of the same size as the product images;
[0014] Calculate a loss function according to the true value and the feature map;
[0015] Back-propagating the product label region segmentation model according to the loss function to calculate the weight update gradient of the model;
[0016] Performing weight update on the product label region segmentation model according to the weight update gradient;
[0017] When the rate of change of the value of the loss function is less than a preset threshold, the training of the product label area segmentation model is completed.
[0018] According to the above aspects and any possible implementation, an implementation is further provided.
[0019] The binary image of the product label area is generated by the following steps:
[0020] Manually labeling the plurality of product images to obtain a label foreground area;
[0021] The captured product image is binarized according to the label foreground area to obtain a binarized image of the product label area.
[0022] According to the above aspects and any possible implementation, an implementation is further provided.
[0023] The product image contains black and white areas; the label segmentation result is a binary image, wherein the label area is white and the non-label area is black;
[0024] The product label area is the white area in the binary image;
[0025] Extracting a product label area from the product image according to the label segmentation result to obtain a product label area image includes:
[0026] The product image is cropped according to the white area in the label segmentation result to obtain a product label area image.
[0027] According to the above aspects and any possible implementation, an implementation is further provided.
[0028] The product image does not contain any black or white area; the label segmentation result is a binary image, wherein the label area is white and the non-label area is black;
[0029] Extracting a product label area from the product image according to the label segmentation result to obtain a product label area image includes:
[0030] Performing a multiplication operation on the product image and the binary image to obtain a label area;
[0031] Image extraction is performed on the product image according to the label area to obtain a product label area image.
[0032] According to the above aspect and any possible implementation manner, an implementation manner is further provided, wherein a multiplication operation is performed on the product image and the binary image to obtain a label area, including:
[0033] Extracting the RGB value of each pixel of the product image;
[0034] Multiplying the RGB value of each pixel of the product image with the RGB value of the pixel at the corresponding position in the binary image;
[0035] The area where the RGB value is not 0 after the multiplication operation is marked as the label area.
[0036] According to the above aspects and any possible implementation, an implementation is further provided.
[0037] The commodity image is at least one frame of image extracted from at least one shot video of the commodities on the commodity shelf, and / or at least one shot image of the commodities on the commodity shelf.
[0038] According to a second aspect of the present disclosure, a device for extracting product label area images based on deep learning is provided. The device comprises:
[0039] A data receiving module, used for receiving product images;
[0040] A region segmentation module is used to input the product image into a pre-trained product label region segmentation model to obtain a label segmentation result of the product image; the product label region segmentation model includes a downsampling feature extraction part, an upsampling feature fusion part and a pyramid pooling module, and the pyramid pooling module is configured with an intermediate convolution layer so that the intermediate convolution layer upsamples the bottom feature map of the downsampling feature extraction part to obtain a target feature map with the same size as the feature map corresponding to each level of the upsampling feature fusion part, so that the product label region segmentation model splices the target feature map with the feature map corresponding to each level of the upsampling feature fusion part;
[0041] The area extraction module is used to extract the product label area from the product image according to the label segmentation result to obtain a product label area image; the product label area includes at least one of a price area, a warranty period area and a label defect area.
[0042] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.
[0043] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.
[0044] The method and device for extracting product label area images based on deep learning provided by the embodiments of the present disclosure include inputting the received product image into a pre-trained product label area segmentation model to obtain a label segmentation result; extracting the product label area from the product image according to the label segmentation result to obtain a product label area image. Among them, the product label area segmentation model adds a pyramid pooling module based on the Unet model structure, which is configured with an intermediate convolution layer so that the intermediate convolution layer upsamples the bottom feature map of the model downsampling feature extraction part to obtain a target feature map with the same size as the feature map corresponding to each level of the upsampling feature fusion part, so that the model splices the target feature map with the feature map corresponding to each level of the upsampling feature fusion part. In this way, the extraction effect of the region of interest (product label area) is effectively improved. Moreover, the product label area segmentation model designed in this way, through multi-scale up and down sampling channels, fuses feature information of different levels, and multiplexes the larger scale spatial structure information and the smaller scale high-level semantic information, has better robustness, and has better segmentation effect for highlights and shadows generated by illumination.
[0045] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:
[0047] Figure 1 A flowchart of a method for extracting a product label area image based on deep learning according to an embodiment of the present disclosure is shown;
[0048] Figure 2 A block diagram of a device for extracting product label area images based on deep learning according to an embodiment of the present disclosure is shown;
[0049] Figure 3 A schematic block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown;
[0050] Figure 4 A schematic diagram of the structure of a product label region segmentation model according to an embodiment of the present disclosure is shown;
[0051] Figure 5 A schematic diagram of the structure of a pyramid pooling module in a product label region segmentation model according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0053] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0054] Figure 1 The flowchart of a method 100 for extracting a product label area image based on deep learning according to an embodiment of the present disclosure is shown. The method 100 includes:
[0055] Step 110: receiving a product image.
[0056] In some embodiments, the product image may be at least one frame of image extracted from at least one video of the products on the product shelf, and / or at least one captured image of the products on the product shelf. For example, a frame of captured image of a wine bottle body is used to extract the product label area image (at least one of the price area image, the warranty period area image, and the label defect area image) contained in the wine bottle body. It can realize the rapid positioning of the area of interest of the label printed on the surface of the product, and by separating the area of interest in the product label from the background, it is convenient for subsequent label content recognition, defect detection, etc.
[0057] Step 120: input the product image into a pre-trained product label region segmentation model to obtain a label segmentation result of the product image.
[0058] In some embodiments, Figure 4The structural diagram of the product label region segmentation model shown in the figure mainly consists of four parts: (1) downsampling feature extraction part, (2) upsampling feature fusion part, (3) spatial attention module, and (4) pyramid pooling module. The product label region segmentation model adds a pyramid pooling module based on the structure of the Unet model. The pyramid pooling module is configured with an intermediate convolution layer so that the intermediate convolution layer upsamples the bottom feature map of the downsampling feature extraction part to obtain a target feature map with the same size as the feature map corresponding to each level of the upsampling feature fusion part, so that the product label region segmentation model can splice the target feature map with the feature map corresponding to each level of the upsampling feature fusion part.
[0059] (1) The downsampling feature extraction part consists of four feature extraction modules of different scales. Each feature extraction module contains two convolution layers (conv) with a convolution kernel of 3×3, and a maximum pooling layer (maxpool) with a kernel of 2×2 and a stride of 2, which is 2x downsampling in the figure. The first feature extraction module contains two convolution layers with 32 and 64 convolution channels respectively, and then a maximum pooling layer is used to reduce the feature map size to 1 / 2 of the original. The next three feature extraction modules are similar to the first one. Based on the previous module, the number of convolution channels of the convolution layer is doubled, and the feature map size is halved. The final feature map size is 1 / 8 of the original size, and the number of channels is 256. Then the upsampling stage begins.
[0060] (2) The upsampling feature fusion part has a mirror image structure to the downsampling part. The difference is that in the upsampling process, the attention map generated by the spatial attention module described in (3) in each downsampling stage will also be fused; the last two upsampling stages will also additionally fuse the feature maps generated by the pyramid pooling module described in (4). Each upsampling stage contains two 3×3 convolutional layers and one deconvolution layer (up-conv). The first convolution layer of each stage will also copy and paste the feature map of its mirrored downsampling stage together, and then perform convolution. The deconvolution layer will upsample the feature map size to 2 times. After the entire upsampling stage, the size of the final output feature map will be restored to the size of the original image.
[0061] (3) Spatial attention module. Figure 4As shown in the yellow rectangle in the figure, it exists in the three different sizes of downsampling stages. It consists of a convolution layer with a convolution kernel of 3×3 and a channel number of 1 and a sigmoid activation function transformation. The feature map is compressed to 1×H×W (height×width) through the convolution layer, and then the value of the feature map is mapped to between 0 and 1 through the sigmoid transformation to represent the size of attention at different positions in space. The output attention map is point-to-point multiplied with the feature map in the upsampling process to amplify the weight of the area of interest in the feature map and reduce the response value of the non-interesting area.
[0062] (4) Pyramid pooling module. Figure 4 It is marked with a red rectangle, and its specific structure is as follows Figure 5 As shown in the figure, the feature map is first subjected to 4 different sizes of maximum pooling to generate 4 sizes of feature maps, and then connected to a 3×3 convolution layer (an additional intermediate convolution layer), and then upsampled to upsample the feature maps of different sizes to the same size, and then concatenated in the channel dimension to obtain the output feature map. Figure 4 In the above, the output of the pyramid pooling module is upsampled twice and then spliced with the feature map of the corresponding upsampling stage. Its function is to further fuse the semantic information of different resolutions on the feature map with rich semantic information, and then introduce this part of the feature map with rich semantic information into different upsampling stages to make up for the loss of semantic information after upsampling. In other words, the proposed pyramid pooling module is different from the general spatial pyramid pooling. It adds an additional intermediate convolution layer and a unique output splicing method (the output of the pyramid pooling module is upsampled twice and then spliced with the feature map of the corresponding upsampling stage), which can more effectively extract the features of regions of interest of different sizes.
[0063] That is, the proposed deep convolutional neural network, through multi-scale up and down sampling pathways, fuses feature information of different levels, and multiplexes the larger-scale spatial structure information and the smaller-scale high-level semantic information. By automatically learning features from the data, compared with the traditional algorithm that manually designs thresholds or features for segmentation, it has better robustness and better segmentation effects for highlights and shadows generated by lighting. The proposed spatial attention mechanism increases the network's weight for the region of interest in forward reasoning, suppresses the response of the non-region of interest, and effectively improves the extraction effect of the region of interest.
[0064] In some embodiments, the pre-training process of the product label area segmentation model is completed by the following steps: obtaining a binary image of the product label area marked in multiple product images as the true value; inputting the multiple product images into the convolutional neural network for forward propagation, and outputting a feature map of the same size as the product image after downsampling and upsampling; calculating the loss function according to the true value and the feature map; performing back propagation on the product label area segmentation model according to the loss function to calculate the weight update gradient of the model; performing weight update on the product label area segmentation model according to the weight update gradient; when the change rate of the value of the loss function is less than a preset threshold, the product label area segmentation model training is completed. The binary image of the marked product label area is generated by the following steps: manually marking the multiple product images to obtain the label foreground area; binarizing the product images according to the label foreground area to obtain the binary image of the marked product label area. For example, ten images of the bottle label of a wine bottle at the same position are obtained. Then the area of the label foreground is segmented and marked by human eyes, and then a binary image is generated according to the marked foreground area. Input the image of the bottle label Figure 4 The convolutional neural network described in the previous section is forward propagated. After the downsampling and upsampling stages, the feature map of the same size as the original image is finally output. The output feature map and the manually annotated binary image (as the true value image) are used to calculate the loss function. The more similar the output feature map is to the manually annotated true value image, the smaller the value of the loss function is, indicating that the network training result is better. Backward propagation is performed according to the loss function to calculate the update gradient of each weight of the network. According to the calculated gradient, the network weights are updated. After a certain number of iterative updates, the value of the loss function tends to be stable. At this time, the network is trained and its network weight parameters are saved. When used later, you only need to input the bottle body image and perform a forward propagation to obtain the binary segmentation result inferred by the network.
[0065] Step 130 : extracting a product label region from the product image according to the label segmentation result to obtain a product label region image.
[0066] In some embodiments, the product label area includes at least one of a price area, a warranty period area, and a label defect area.
[0067] In some embodiments, the product image contains black and white areas; the label segmentation result is a binary image; wherein the label area is white and the non-label area is black; the product label area is the white area in the binary image; according to the label segmentation result, the product label area is extracted from the product image to obtain a product label area image, including: cropping the product image according to the white area in the label segmentation result to obtain a product label area image.
[0068] In some embodiments, the product image does not contain black and white areas; the label segmentation result is a binary image (only two colors, black and / or white); wherein the label area is white and the non-label area is black (where black=0 and white=1); extracting the product label area from the product image according to the label segmentation result to obtain a product label area image includes: multiplying the product image with the binary image to obtain the label area; performing image extraction on the product image according to the label area to obtain a product label area image. Multiplying the product image with the binary image to obtain the label area includes: extracting the RGB value of each pixel of the product image; multiplying the RGB value of each pixel of the product image with the RGB value of the pixel at the corresponding position in the binary image; marking the area with a non-0 RGB value after the multiplication as the label area.
[0069] The method proposed in the present invention has good versatility and can be applied to any product label. It is not affected by differences in the appearance and shape of the product. It only needs to replace the label data and tune the model to adapt to the corresponding product label and extract the image of the corresponding area.
[0070] On this basis, during the training process, the product label area image can be manually labeled in advance to determine the type of product, such as price label, warranty period label or defective label, etc., so that machine learning can achieve automatic classification. Combined with the identification conditions, it can automatically determine whether the product is expired, etc., which can be expanded and refined on the basis of the above method described in the present disclosure, and no more examples are given here.
[0071] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.
[0072] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.
[0073] Figure 2 FIG. 2 shows a block diagram of a product label area image extraction device 200 based on deep learning according to an embodiment of the present disclosure. Figure 2 As shown, the device 200 includes:
[0074] The data receiving module 210 is used to receive product images;
[0075] A region segmentation module 220 is used to input the product image into a pre-trained product label region segmentation model to obtain a label segmentation result of the product image; the product label region segmentation model includes a downsampling feature extraction part, an upsampling feature fusion part and a pyramid pooling module, and the pyramid pooling module is configured with an intermediate convolution layer so that the intermediate convolution layer upsamples the bottom feature map of the downsampling feature extraction part to obtain a target feature map with the same size as the feature map corresponding to each level of the upsampling feature fusion part, so that the product label region segmentation model splices the target feature map with the feature map corresponding to each level of the upsampling feature fusion part;
[0076] The region extraction module 230 is used to extract the product label region from the product image according to the label segmentation result to obtain a product label region image; the product label region includes at least one of a price region, a warranty period region and a label defect region.
[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0078] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.
[0079] Figure 3 A schematic block diagram of an electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0080] The electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a ROM 302 or a computer program loaded from a storage unit 308 into a RAM 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.
[0081] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0082] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).
[0083] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0084] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0085] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0086] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0087] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0088] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0089] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0090] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for extracting product label area images based on deep learning, characterized in that: include: Receiving product images; The product image is input into a pre-trained product label region segmentation model to obtain a label segmentation result of the product image; the product label region segmentation model includes a downsampling feature extraction part, an upsampling feature fusion part and a pyramid pooling module, and the pyramid pooling module is configured with an intermediate convolution layer, so that the intermediate convolution layer upsamples the bottom feature map of the downsampling feature extraction part to obtain a target feature map with the same size as the feature map corresponding to each level of the upsampling feature fusion part, so that the product label region segmentation model splices the target feature map with the feature map corresponding to each level of the upsampling feature fusion part; The product label area is extracted from the product image according to the label segmentation result to obtain a product label area image; the product label area includes at least one of a price area, a warranty period area and a label defect area.
2. The method according to claim 1, characterized in that The product label region segmentation model also includes a convolutional neural network; The pre-training process of the product label region segmentation model is completed through the following steps: Obtain a binary image of the product label area of the multiple product images as the true value; Input the multiple product images into a convolutional neural network for forward propagation, and after downsampling and upsampling, output feature maps of the same size as the product images; Calculate a loss function according to the true value and the feature map; Back-propagating the product label region segmentation model according to the loss function to calculate the weight update gradient of the model; Performing weight update on the product label region segmentation model according to the weight update gradient; When the rate of change of the value of the loss function is less than a preset threshold, the training of the product label area segmentation model is completed.
3. The method according to claim 2, characterized in that The binary image of the product label area is generated by the following steps: Manually labeling the plurality of product images to obtain a label foreground area; The captured product image is binarized according to the label foreground area to obtain a binarized image of the product label area.
4. The method according to claim 1, characterized in that: The product image contains black and white areas; the label segmentation result is a binary image, wherein the label area is white and the non-label area is black; The product label area is the white area in the binary image; Extracting a product label area from the product image according to the label segmentation result to obtain a product label area image includes: The product image is cropped according to the white area in the label segmentation result to obtain a product label area image.
5. The method according to claim 1, characterized in that: The product image does not contain any black or white area; the label segmentation result is a binary image, wherein the label area is white and the non-label area is black; Extracting a product label area from the product image according to the label segmentation result to obtain a product label area image includes: Performing a multiplication operation on the product image and the binary image to obtain a label area; Image extraction is performed on the product image according to the label area to obtain a product label area image.
6. The method according to claim 5, characterized in that The product image and the binary image are multiplied to obtain a label area, including: Extracting the RGB value of each pixel of the product image; Multiplying the RGB value of each pixel of the product image with the RGB value of the pixel at the corresponding position in the binary image; The area where the RGB value is not 0 after the multiplication operation is marked as the label area.
7. The method according to claim 1, characterized in that The commodity image is at least one frame of image extracted from at least one shot video of the commodities on the commodity shelf, and / or at least one shot image of the commodities on the commodity shelf.
8. A product label area image extraction device based on deep learning, characterized in that: include: A data receiving module, used for receiving product images; A region segmentation module is used to input the product image into a pre-trained product label region segmentation model to obtain a label segmentation result of the product image; the product label region segmentation model includes a downsampling feature extraction part, an upsampling feature fusion part and a pyramid pooling module, and the pyramid pooling module is configured with an intermediate convolution layer so that the intermediate convolution layer upsamples the bottom feature map of the downsampling feature extraction part to obtain a target feature map with the same size as the feature map corresponding to each level of the upsampling feature fusion part, so that the product label region segmentation model splices the target feature map with the feature map corresponding to each level of the upsampling feature fusion part; The area extraction module is used to extract the product label area from the product image according to the label segmentation result to obtain a product label area image; the product label area includes at least one of a price area, a warranty period area and a label defect area.
9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.