Ice cream detection and identification method, system and device for market audit

Abstract

The present application relates to an ice cream detection and identification method, system and device for market inspection, which includes obtaining a captured image of ice cream in a freezer, preprocessing the captured image, and sending the captured image into a neural network model; extracting, processing and calculating the image information through the ice cream detection neural network model to obtain a number of independent rectangular images surrounding the target; recognizing the rectangular image by establishing a multi-class ice cream recognition model, outputting the prediction result, fusing the prediction result to obtain the recognition result, and combing out the structured information according to the recognition result; detecting whether a new captured image appears, and repeating the above ice cream detection and identification process for the new captured image. The present application has the effect of realizing automatic detection of ice cream and identifying structured information, thereby improving the efficiency of market inspection work.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to an ice cream detection and identification method, system and device for market audit. Background Art

[0002] With the development of the economy, household income and household consumption levels have increased year by year. The market size of the ice cream industry has grown rapidly, maintaining a steady growth rate of nearly 5% each year. The size of China's ice cream market ranks first in the world. It is estimated that the market size of my country's ice cream will reach at least 160 billion yuan in 2021. The expansion of the market has also given brands more development opportunities. The ice cream market has become a battleground for major businesses.

[0003] The information collection of the ice cream market is an important basic work for establishing an ice cream marketing system. The collection, analysis and use of market information drives the development of key marketing businesses such as ice cream demand forecasting, supply organization, supply supply and brand cultivation. It is an important means for the ice cream industry to develop new products, grasp market trends, understand the situation of competitors and formulate sales plans.

[0004] Traditionally, market inspections of ice cream, from counting the number and types of ice cream to compiling structured information, are all done manually, which is very inefficient. There is still no better solution for automated detection and identification of ice cream. Summary of the invention

[0005] In order to automatically detect ice cream and identify structured information, the present application provides an ice cream detection and identification method, system and device for market audit.

[0006] In a first aspect, the present application provides an ice cream detection and identification method for market audit, which adopts the following technical solution:

[0007] Obtaining a collected image of ice cream in a freezer, preprocessing the collected image, and sending the collected image to a neural network model;

[0008] The image information is extracted, processed and calculated through the ice cream detection neural network model to obtain several independent rectangular images surrounding the target.

[0009] By establishing a multi-class ice cream recognition model to recognize rectangular images, the prediction results are output, the prediction results are fused to obtain the recognition results, and the structured information is sorted out according to the recognition results;

[0010] Detect whether there is a new collected image, and repeat the above ice cream detection and recognition process for the new collected image.

[0011] By adopting the above technical solution, after the collected image is subjected to feature extraction detection by the ice cream detection neural network model and the ice cream part in the image is cut out, the obtained rectangular image is automatically sent to the ice cream recognition model for type recognition, and the prediction result is output. Then, the recognition result is obtained through information fusion, and the structured information is sorted out according to the recognition structure for automatic statistics, thereby realizing the automatic detection and recognition of ice cream information. In the whole process, only the collection of the collected image needs to be performed manually, which greatly reduces the labor cost. At the same time, it can also speed up the collection of feedback ice cream market information, grasp the market dynamics, dynamically formulate marketing strategies, and improve economic benefits.

[0012] Optionally, the ice cream detection neural network model adopts a neural network model based on yolov5, and the ice cream recognition model adopts a recognition model based on mobilenet+arcsoftmaxloss.

[0013] By adopting the above technical solutions, the neural network model based on yolov5 has the characteristics of fast detection speed, and the mobilenet+arcsoftmaxloss recognition model has the characteristics of high accuracy.

[0014] Optionally, the step of extracting, processing and calculating the image information by using the ice cream detection neural network model to obtain a plurality of rectangular images independently surrounding the ice cream image includes:

[0015] The preprocessed captured image is sent to a trained ice cream detection neural network model with instance segmentation function to output the confidence that each grid contains a target and the relevant information corresponding to the grid, wherein the relevant information includes the target center coordinate information and the corresponding boundary point coordinate information of a preset fixed number P that surrounds the target;

[0016] Determine in turn whether the confidence level of each grid containing the target is greater than the set threshold;

[0017] If there is a grid containing a target whose confidence level is greater than the set threshold, the relevant information corresponding to the grid is retained;

[0018] Determine the polygonal frame of the target based on the target center coordinate information and its corresponding boundary point coordinate information, and then determine the minimum circumscribed rectangular frame of the ice cream based on the polygonal frame;

[0019] The image within the minimum circumscribed rectangular frame is transformed into a standard image of a preset standard size by using an affine transformation formula;

[0020] The objects in the polygonal frame area of ​​the standard image are retained, and the pixel values ​​outside the polygonal frame area are uniformly set to preset values ​​to obtain a rectangular image.

[0021] By adopting the above technical solution, due to the different placement shapes of ice cream, the traditional yolov5-based neural network comes with a detection anchor frame, which can be used directly, but the detection anchor frame will also surround a large number of non-ice cream parts, especially when the ice cream is tilted. The non-ice cream part surrounded by the detection anchor frame will be larger than the ice cream part, which seriously interferes with the recognition of the subsequent ice cream recognition model and is prone to errors. Therefore, the present application is provided with a number of boundary point coordinate information that surrounds the entire target, and it is fused into the yolov5 neural network as a feature for training and improvement, so that the collected image can accurately identify the boundary points surrounding the ice cream part after passing through the ice cream detection neural network model, and then connect the boundary points to form an irregular polygonal frame, and then determine the smallest circumscribed rectangular frame according to the size of the polygonal frame, which can make the segmentation of the ice cream image more accurate. At the same time, the polygonal frame is also used to filter the background of the non-ice cream part in the circumscribed rectangular frame, which can effectively improve the recognition accuracy in the subsequent ice cream recognition model.

[0022] Optionally, the affine transformation formula includes:

[0023]

[0024] Affine transformation matrix The parameters in are specifically solved as follows: a superlinear equation group is established through the aspect ratio of the ice cream structure before and after the affine transformation and its coordinate point information, and the parameter values ​​of the affine matrix are estimated.

[0025] The rectangular image coordinates obtained by the affine formula can reduce errors and improve accuracy.

[0026] Optionally, the ice cream detection neural network model includes: an input end, a reference network, a Neck network and a Head output end;

[0027] The input end is used to normalize the collected image and scale it to a size of 608*608 before feeding it into the benchmark network;

[0028] The baseline network is used to extract features. It downsamples layer by layer to obtain feature layers of 76*76, 38*38, and 19*19 sizes, and sends the feature layer of 19*19 size to the Neck network.

[0029] Neck network is used to connect and restore the 19*19 feature layer with the features of the same size on the backbone through upsampling to obtain 38*38 and 76*76 feature layers respectively. The obtained 19*19, 38*38 and 76*76 feature layers are sent to the Head output end, wherein the grid containing the target center coordinates on each feature layer is generated with respect to the x offset and y offset of the upper left corner coordinate of the grid. At the same time, the x offset and y offset of the boundary points with a fixed number P surrounding the target on the feature layer are generated with respect to the upper left corner coordinate of the grid. In addition, the confidence of the target is also generated, with a total of 2+P*2+1 upper thresholds. Let the feature layer size of each layer output be S*S, then the feature layer output of each layer is S*S*(2+P*2+1);

[0030] The head output end is used to send the feature layers of 19*19, 38*38 and 76*76 sizes into the convolution kernel to generate the feature image, and then send it to the fully connected layer for full connection, to obtain and output the confidence of each grid containing the target, the target center coordinate information and the corresponding boundary point coordinate information with a fixed number P that surrounds the target.

[0031] By adopting the above technical solution, the output amount of the output feature layer generated in the Neck network is determined by a fixed number P. The user can adjust and set it according to the requirements of the recognition accuracy of the boundary points of the ice cream. At the same time, the overall detection processing speed can also be adjusted.

[0032] Optionally, the fixed number P is 20.

[0033] By adopting the above technical solution, the fixed number P is set to 20, which can achieve the optimal detection processing speed without affecting the accuracy of the boundary area detection of the ice cream.

[0034] Optionally, the ice cream detection neural network model training method includes:

[0035] Obtaining a collection image and a sample image of ice cream in a freezer, wherein the sample image has been annotated with various label formats in advance;

[0036] The collected image is sent to the ice cream detection neural network model to obtain training data with the same label format as the sample image, the training data and the label data are brought into the loss function to calculate the loss, and then the loss is propagated back through the back propagation algorithm to update the parameters of the ice cream detection neural network model. This process is repeated continuously to finally obtain a trained parameter matrix, and the parameter matrix is ​​loaded into the ice cream detection neural network model for updating to complete the training of the ice cream detection neural network model.

[0037] By adopting the above technical solution, a neural network is used to detect and segment the image of ice cream, and the detected ice cream contour is compared with the real contour. Then, the parameters of the neural network are adjusted, and the ice cream detection neural network model is trained to enable the ice cream detection neural network model to accurately detect and obtain the ice cream contour, thereby effectively improving the detection accuracy.

[0038] Optionally, the loss function includes:

[0039]

[0040]

[0041]

[0042] Loss = lobj + loff + lpoly;

[0043] In the formula, lobj represents the loss of whether there is a target, loff represents the loss of the center point coordinate offset, and lpoly represents the loss of the boundary point coordinate relative to the center point; λ obj ,λ nobj ,λ off ,λ poly They represent the weight coefficients of the corresponding losses respectively; S represents the side length of the feature map, and P represents the number of boundary point coordinates; Indicates whether there is an object in the i-th grid on the feature map, if yes, it is 1, if not, it is 0; Indicates whether there is no target in the i-th grid on the feature map, if not, it is 1, if it is, it is 0; c i , They respectively represent the true value and predicted value of whether the i-th grid of the feature map has an object; x i ,y i Respectively represent the actual offset of the target in the ith grid relative to the upper left corner of the grid in the x and y directions, Respectively represent the predicted offset of the target in the ith grid to the upper left corner of the grid in the x and y directions, which are used to determine the actual coordinates of the target center point; smooth l1 represents a regression loss function; x i,j ,y i,j They represent the actual offsets of the coordinates of the P boundary points relative to the upper left corner of the grid in the x and y directions, respectively. They respectively represent the values ​​of the predicted offsets of the coordinates of the P boundary points relative to the upper left corner of the grid in the x and y directions, which are used to determine the bounding box surrounding the target; Loss represents the sum of the loss functions.

[0044] By adopting the above technical solution, the smooth l1 The regression loss function has good stability and robustness. The confidence loss is calculated by the lobj calculation formula, the center point coordinate offset loss is calculated by the loff calculation formula, and the coordinate offset loss of each boundary point is calculated by the lpoly calculation formula. The offsets are summed to get the final offset, and the error between the ice cream contour and the true contour can be obtained. The parameters of the neural network are adjusted according to the error, so that the neural network can get more accurate results in the next round of training.

[0045] Optionally, the step of preprocessing the acquired image includes:

[0046] Determine whether the quality of the acquired image is qualified;

[0047] When the image quality is unsatisfactory, the image is recaptured;

[0048] When the image quality is qualified, the collected images are uniformly scaled to a size of 608*608, and the collected images are detected and corresponding data enhancement operations are performed, including up and down flipping, Gaussian blur, rotation, simulated occlusion, and simulated glass reflection.

[0049] By adopting the above technical solution, the quality of the collected image can be identified, which can reduce the situation of inaccurate recognition, and at the same time, can reduce the processing error and the impact on the image processing efficiency. Further data enhancement operations are performed on qualified collected images, which can make the quality of the image easier to identify and improve the accuracy of the detection effect.

[0050] In the second aspect, the present application provides an ice cream detection and identification system for market audit, which adopts the following technical solution:

[0051] An ice cream detection and identification system for market audit, comprising: an image acquisition module, for acquiring an image of ice cream in a freezer, preprocessing the image, and sending the image into a neural network model;

[0052] An image processing module is used to extract, process and calculate image information through an ice cream detection neural network model to obtain a number of independent rectangular images surrounding the ice cream image;

[0053] An image recognition module is used to recognize rectangular images by establishing a multi-class ice cream recognition model, output prediction results, fuse the prediction results to obtain recognition results, and sort out structured information based on the recognition results;

[0054] The re-detection module is used to detect whether a new collected image appears and repeat the above ice cream detection and recognition process for the new collected image.

[0055] In a third aspect, the present application provides an electronic device, which adopts the following technical solution:

[0056] An electronic device comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and execute the above-mentioned ice cream detection and identification method for market inspection.

[0057] In a fourth aspect, the present application provides a computer storage medium, which adopts the following technical solution:

[0058] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute the above-mentioned ice cream detection and identification method for market audit.

[0059] In summary, the present application includes at least one of the following beneficial technical effects:

[0060] After the collected image is subjected to feature extraction detection by the ice cream detection neural network model and the ice cream part in the image is cut out, the obtained rectangular image is automatically sent to the ice cream recognition model for type recognition, and the prediction result is output. The recognition result is then obtained through information fusion, and the structured information is sorted out according to the recognition structure for automatic statistics, thus realizing the automatic detection and recognition of ice cream information. In the whole process, only the collection of the collected image needs to be performed manually, which greatly reduces the labor cost. At the same time, it can also speed up the collection of feedback ice cream market information, grasp the market dynamics, dynamically formulate marketing strategies, and improve economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of an ice cream detection and identification method for market inspection according to one embodiment of the present application.

[0062] Figure 2 It is a flowchart of the specific steps of preprocessing the captured image in step S1 in another embodiment of the present application.

[0063] Figure 3 This is a specific step flow chart of step S2 in another embodiment of the present application.

[0064] Figure 4 It is a structural diagram of an ice cream detection neural network model in another embodiment of the present application.

[0065] Figure 5 It is a flowchart of the specific steps of the ice cream detection neural network model training method in another embodiment of the present application.

[0066] Figure 6 It is a structural diagram of an ice cream detection and identification system for market inspection according to one embodiment of the present application.

[0067] In the figure, 1. Image acquisition module; 2. Image processing module; 3. Image recognition module; 4. Re-detection module. DETAILED DESCRIPTION

[0068] The following is combined with Figure 1-Figure 6 This application is described in further detail.

[0069] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-Figure 6 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] Reference Figure 1 The present application embodiment discloses an ice cream detection and identification method for market audit, comprising the following steps:

[0071] S1. Obtain a captured image of ice cream in a freezer, pre-process the captured image, and send the captured image to a neural network model.

[0072] Specifically, the collected image is obtained by shooting with a camera, and the collected image is pre-processed to determine whether the collected image is qualified, and a series of operations such as data enhancement are performed, and then the collected image is sent to the neural network.

[0073] in, Figure 2 This is a flowchart of an optional implementation method for preprocessing the collected image in step S1 disclosed in this application, refer to Figure 2 , step S1 specifically includes the following steps S10, S11 and S12:

[0074] S10: Determine whether the quality of the acquired image is qualified.

[0075] S11. When the image quality is unqualified, re-capture the image.

[0076] S12. When the image quality is qualified, the collected images are uniformly scaled to a size of 608*608, and the collected images are detected and corresponding data enhancement operations are performed. The data enhancement operations include up and down flipping, Gaussian blur, rotation, simulated occlusion, and simulated glass reflection.

[0077] Among them, judging whether the quality of the collected image is qualified can be achieved by detecting whether the image clarity is greater than a preset threshold. Specifically, an optional implementation method is: since the more high-frequency components in the image, the greater the difference between the sudden pixel and the adjacent pixel, that is, the greater the clarity, therefore, the two grayscale differences in the right area of ​​each pixel are subtracted and multiplied, and then accumulated pixel by pixel to calculate the square of the grayscale difference between two adjacent pixels, as follows:

[0078] D(f)=∑ y ∑ x |f(x+2,y)-f(x,y)| 2 ;

[0079] Where: f(x,y) represents the grayscale value of the pixel (x,y) corresponding to image f, and D(f) is the image definition. This method is very efficient, and the calculation of 10 images with a resolution of 1280*1024 can be completed in less than 70 milliseconds.

[0080] In addition, when the image quality is qualified, the collected images are uniformly scaled to a size of 608*608, and then the collected images are automatically detected and data enhancement operations such as up and down flipping, Gaussian blur, rotation, simulated occlusion, and simulated glass reflection are performed. Among them, the data enhancement for simulated occlusion and glass reflection is due to the complexity and changeability of the actual scene. There are also serious occlusions between ice creams in the collected images, and in many cases, there will be glass reflections that cause the ice cream image to be blurred. In order to solve the above problems and improve the generalization ability of the ice cream detection neural network model, this application simulates these scenes by means of map fusion, simulates the glass reflection effect by image fusion, and simulates the occlusion effect by taking down the ice cream image in the freezer and then randomly erasing certain parts of the above image, and then pasting the processed image back. The above method can further increase the amount of data on the basis of the existing amount of data used for model training.

[0081] S2. The image information is extracted, processed and calculated through the ice cream detection neural network model to obtain a number of independent rectangular images surrounding the target.

[0082] Specifically, the detection accuracy of the rectangular image independently surrounding the ice cream calculated by the ice cream detection neural network model with instance segmentation function trained by deep learning is high. The training data of the detection model comes from the collected image pre-processed in step S12. The ice cream detection neural network model can adopt but is not limited to algorithms such as yolo, ssd, and faster-rcnn.

[0083] Among them, in this embodiment, the ice cream detection neural network model preferably adopts a neural network model based on yolov5. Specifically, Figure 3 This is a flowchart of an optional implementation method of step 2 disclosed in this application, refer to Figure 3 , step S2 specifically includes the following steps S20, step S21, step S22, step S23, step S24 and step S25:

[0084] S20. Send the preprocessed captured image to a trained ice cream detection neural network model with instance segmentation function to output the confidence that each grid contains the target and the relevant information corresponding to the grid, wherein the relevant information includes the target center coordinate information and the corresponding boundary point coordinate information of a preset fixed number P that surrounds the target.

[0085] Specifically, the ice cream detection neural network model outputs the confidence that each grid contains the target and the relevant information corresponding to the grid. The relevant information includes the target center coordinate information and its corresponding coordinate information of a preset fixed number P of boundary points that completely surround the target. In this embodiment, the fixed number P can be selected as 20, that is, 20 boundary points are set to completely surround a single ice cream.

[0086] S21. Determine in turn whether the confidence level of each grid containing a target is greater than a set threshold.

[0087] Specifically, starting from the image in the upper left corner of the captured image and moving toward the right, it is determined whether the confidence that each grid contains a target is greater than a set threshold. In this embodiment, the set threshold may be selected as 0.65.

[0088] S22. If the confidence level of a grid containing a target is greater than a set threshold, the relevant information corresponding to the grid is retained.

[0089] S23, determining a polygonal frame of the target according to the target center coordinate information and its corresponding boundary point coordinate information, and then determining a minimum circumscribed rectangular frame of the ice cream according to the polygonal frame.

[0090] Specifically, when there is a grid containing a target and the confidence level is greater than a set threshold, the relevant information corresponding to the grid is retained. At the same time, the polygonal box area of ​​the target is connected in sequence according to the target center coordinate information and its corresponding boundary coordinate information, and then the length and width of the rectangular box are determined according to the maximum diameter of the polygonal box in the x and y directions, thereby determining the minimum circumscribed rectangular box.

[0091] S24, transforming the image within the minimum circumscribed rectangular frame into a standard image of a preset standard size using an affine transformation formula.

[0092] Specifically, the affine transformation formula includes:

[0093]

[0094] That is, x1 = a 11 x0+a 12 x0+b1,y1=a 21 x0+a 22 x0+b1.

[0095] Where: (x0, y0) is the coordinate of the point on the image within the minimum bounding rectangle before transformation, (x1, y1) is the coordinate of the point on the image within the minimum bounding rectangle after transformation, and the affine transformation matrix is The parameters in are specifically solved as follows: a superlinear equation system is established through the aspect ratio of the rectangular image before and after the affine transformation and its coordinate point information, and then the parameter values ​​of the affine matrix are estimated based on methods such as least squares or SVD decomposition.

[0096] S25, retaining the target in the polygonal frame area in the standard image, and setting the pixel values ​​outside the polygonal frame area to a preset value, so as to obtain a rectangular image.

[0097] Specifically, a polygonal box is used to unify the pixels of the standard image except for the ice cream. In this embodiment, the preset value is set to 0, that is, the image except the ice cream part is set to a black background, which forms a strong pixel difference with the ice cream part, which is beneficial to improving the recognition accuracy of the multi-category ice cream recognition model.

[0098] S3. Recognize the rectangular image by establishing a multi-class ice cream recognition model, output the prediction result, fuse the prediction result to obtain the recognition result, and sort out the structured information based on the recognition result.

[0099] Specifically, the ice cream recognition model adopts a recognition model based on mobilenet+arcsoftmaxloss, where mobilenet is a neural network, arcsoftmaxloss is a loss function used in the neural network training process, and the recognition model based on mobilenet+arcsoftmaxloss is a conventional technology in the field. This application can adopt but is not limited to the above neural network and can also use other neural networks. After the ice cream part of the captured image is segmented, the ice cream recognition model is further used to identify the type of ice cream, thereby outputting a predicted structure, and the prediction result is information fused to obtain a recognition result, and the structured information is sorted out according to the recognition result.

[0100] S4. Detect whether there is a new collected image, and repeat the above ice cream detection and recognition process for the new collected image.

[0101] Specifically, the system automatically detects whether a new collected image appears. If a new collected image appears, the above ice cream detection and recognition process is repeated.

[0102] In addition, refer to Figure 4 The ice cream detection neural network model in step 2 disclosed in the present application includes an input end, a reference network, a Neck network and a Head output end.

[0103] The input end is used to normalize the collected image and scale it to a size of 608*608, and then standardize it with a mean of 0.45 and a variance of 0.3 before sending it to the benchmark network.

[0104] The baseline network is used to extract features. It obtains feature layers of 76*76, 38*38, and 19*19 sizes layer by layer through downsampling, and sends the feature layer of 19*19 size to the Neck network.

[0105] Neck network is used to connect and restore the 19*19 feature layer with the features of the same size on the backbone through upsampling to obtain 38*38 and 76*76 feature layers respectively. The obtained 19*19, 38*38 and 76*76 feature layers are sent to the head output end. For each feature layer, the grid containing the target center coordinates is generated with respect to the x offset and y offset of the upper left corner coordinate of the grid. At the same time, the x offset and y offset of the boundary points with a fixed number P surrounding the target on the feature layer are generated with respect to the upper left corner coordinate of the grid. In addition, the confidence of the target is also generated, and a total of 2+P*2+1 upper thresholds can be generated. The feature layer size of each layer output is S*S, and the feature layer output of each layer is S*S*(2+P*2+1).

[0106] The head output end is used to send the feature layers of 19*19, 38*38 and 76*76 sizes into the convolution kernel to generate the feature image, and then send it to the fully connected layer for full connection, to obtain and output the confidence of each grid containing the target, the target center coordinate information and the corresponding boundary point coordinate information with a fixed number P that surrounds the target.

[0107] Specifically, the ice cream detection neural network model is an improved model based on yolov5, in which the CSPDarknet53 structure + Focus structure is used as the baseline network; the Neck network is located between the baseline network and the Head output end. The Neck network can further improve the diversity and robustness of the features. The Neck network uses the SPP module and the FPN+PAN module to process the feature layer.

[0108] Furthermore, Figure 5 This is a flowchart of an optional implementation method of the ice cream detection neural network model training method of this application, refer to Figure 5 ,include:

[0109] S5. Obtain the collected image and sample image of the ice cream in the freezer. The sample image has been annotated with various label formats in advance.

[0110] Among them, the labelme tool is used to manually annotate the ice cream part in the collected image to obtain a sample image, which is used as the basis for result comparison and parameter adjustment. Specifically, the annotation method is: 20 uniform boundary points are outlined for each ice cream, and the closed area surrounded by the 20 boundary points is the area of ​​the ice cream. By analyzing the boundary points of each ice cream on each picture, the mapping of each ice cream on three feature layers is obtained. The sizes of the three feature layers are 19*19, 38*38 and 76*76 respectively. The mapping parameters include the confidence of the ice cream and the x offset and y offset of each boundary point of each ice cream relative to the upper left corner of the corresponding grid. Regarding the confidence, that is, the value of the position where the center coordinate of each ice cream is mapped to the corresponding coordinate on the feature layer is set to 1, and the rest of the positions are set to 0.

[0111] S50, sending the collected image to the ice cream detection neural network model to obtain training data with the same label format as the sample image, bringing the training data and the label data into the loss function to calculate the loss, and then back-propagating the loss back through the back-propagation algorithm to update the parameters of the ice cream detection neural network model, and repeating this process continuously to finally obtain a trained parameter matrix, load the parameter matrix into the ice cream detection neural network model for updating, and complete the training of the ice cream detection neural network model.

[0112] Specifically, the loss function includes:

[0113]

[0114]

[0115]

[0116] Loss = lobj + loff + lpoly;

[0117] Where: lobj represents the loss of whether there is a target, loff represents the loss of the center point coordinate offset, lpoly represents the loss of the boundary point coordinate relative to the center point; λ obj ,λ nobj ,λ off ,λ poly They represent the weight coefficients of the corresponding losses respectively; S represents the side length of the feature map, and P represents the number of boundary point coordinates; Indicates whether there is an object in the i-th grid on the feature map, if yes, it is 1, if not, it is 0; Indicates whether there is no target in the i-th grid on the feature map, if not, it is 1, if it is, it is 0; c i , They respectively represent the true value and predicted value of whether the i-th grid of the feature map has an object; x i,y i Respectively represent the actual offset of the target in the ith grid relative to the upper left corner of the grid in the x and y directions, Respectively represent the predicted offset of the target in the ith grid to the upper left corner of the grid in the x and y directions, which are used to determine the actual coordinates of the target center point; smooth l1 represents a regression loss function; x i,j ,y i,j They represent the actual offsets of the coordinates of the P boundary points relative to the upper left corner of the grid in the x and y directions, respectively. They respectively represent the values ​​of the predicted offsets of the coordinates of the P boundary points relative to the upper left corner of the grid in the x and y directions, which are used to determine the bounding box surrounding the target; Loss represents the sum of the loss functions.

[0118] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0119] The present application also provides an ice cream detection and identification system for market auditing, and the ice cream detection and identification system for market auditing corresponds one-to-one with the ice cream detection and identification method for market auditing in the embodiment. Figure 6 The ice cream detection and identification system for market inspection includes: an image acquisition module 1, an image processing module 2, an image recognition module and a re-detection module 4. The detailed description of each functional module is as follows:

[0120] The image acquisition module 1 is used to acquire the image of the ice cream in the freezer, pre-process the image, and send the image to the neural network model.

[0121] The image processing module 2 is used to extract, process and calculate the image information through the ice cream detection neural network model to obtain a plurality of independent rectangular images surrounding the ice cream image.

[0122] The image recognition module 3 is used to recognize the rectangular image by establishing a multi-class ice cream recognition model, output the prediction result, fuse the prediction result to obtain the recognition result, and sort out the structured information according to the recognition result.

[0123] The re-detection module 4 is used for detecting whether a new collected image appears, and repeating the above ice cream detection and recognition process for the new collected image.

[0124] Among them, the ice cream detection and recognition system obtains the collected image of the ice cream in the freezer from the camera, and uses the collected image module 1 to pre-process the collected image, and then sends the collected image to the neural network model, and then uses the image processing module 2 to call the ice cream detection neural network model to extract, process and calculate the image information, and obtain a number of independent rectangular images surrounding the ice cream image, and then uses the image recognition module 3 to call the established multi-class ice cream recognition model to recognize the rectangular image, output the prediction result, and fuse the information of the prediction result to obtain the recognition result, and sort out the structured information according to the recognition result. After obtaining the structured information, the re-detection module 4 is used to detect whether there is a new collected image, so as to repeat the above ice cream detection and recognition process for the new collected image.

[0125] For the specific definition of the ice cream detection and identification system for market audit, please refer to the definition of the ice cream detection and identification method for market audit in the context, which will not be repeated here. Each module in the above-mentioned ice cream detection and identification system for market audit can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0126] The embodiment of the present application discloses an electronic device. An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the following steps are performed:

[0127] S1. Obtain a captured image of ice cream in a freezer, pre-process the captured image, and send the captured image to a neural network model.

[0128] S2. The image information is extracted, processed and calculated through the ice cream detection neural network model to obtain a number of independent rectangular images surrounding the target.

[0129] S3. Recognize the rectangular image by establishing a multi-class ice cream recognition model, output the prediction result, fuse the prediction result to obtain the recognition result, and sort out the structured information based on the recognition result.

[0130] S4. Detect whether there is a new collected image, and repeat the above ice cream detection and recognition process for the new collected image.

[0131] The sub-steps of preprocessing and refining the collected image in step S1 include:

[0132] S10: Determine whether the quality of the acquired image is qualified.

[0133] S11. When the image quality is unqualified, re-capture the image.

[0134] S12. When the image quality is qualified, the collected images are uniformly scaled to a size of 608*608, and the collected images are detected and corresponding data enhancement operations are performed. The data enhancement operations include up and down flipping, Gaussian blur, rotation, simulated occlusion, and simulated glass reflection.

[0135] The sub-steps of step S2 include:

[0136] S20. Send the preprocessed captured image to a trained ice cream detection neural network model with instance segmentation function to output the confidence that each grid contains the target and the relevant information corresponding to the grid, wherein the relevant information includes the target center coordinate information and the corresponding boundary point coordinate information of a preset fixed number P that surrounds the target.

[0137] S21. Determine in turn whether the confidence level of each grid containing a target is greater than a set threshold.

[0138] S22. If the confidence level of a grid containing a target is greater than a set threshold, the relevant information corresponding to the grid is retained.

[0139] S23, determining a polygonal frame of the target according to the target center coordinate information and its corresponding boundary point coordinate information, and then determining a minimum circumscribed rectangular frame of the ice cream according to the polygonal frame.

[0140] S24, transforming the image within the minimum circumscribed rectangular frame into a standard image of a preset standard size using an affine transformation formula.

[0141] S25, retaining the target in the polygonal frame area in the standard image, and setting the pixel values ​​outside the polygonal frame area to a preset value, so as to obtain a rectangular image.

[0142] In addition, the ice cream detection neural network model training method in this application includes the following steps:

[0143] S5. Obtain the collected image and sample image of the ice cream in the freezer. The sample image has been annotated with various label formats in advance.

[0144] S50, sending the collected image to the ice cream detection neural network model to obtain training data with the same label format as the sample image, bringing the training data and the label data into the loss function to calculate the loss, and then back-propagating the loss back through the back-propagation algorithm to update the parameters of the ice cream detection neural network model, and repeating this process continuously to finally obtain a trained parameter matrix, load the parameter matrix into the ice cream detection neural network model for updating, and complete the training of the ice cream detection neural network model.

[0145] An embodiment of the present application also discloses a computer-readable storage medium storing a computer program that can be loaded and executed by a processor. When the computer program is executed by the processor, the steps of any of the above-mentioned ice cream detection and identification methods for market audits are implemented and the same effect can be achieved.

[0146] The computer-readable storage medium includes, for example, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0147] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. An ice cream detection and identification method for market audit, characterized in that: include: Obtaining a collected image of ice cream in a freezer, preprocessing the collected image, and sending the collected image to a neural network model; The ice cream detection neural network model adopts a neural network model based on yolov5, and the ice cream recognition model adopts a recognition model based on mobilenet+arcsoftmaxloss; The ice cream detection neural network model extracts, processes and calculates the image information to obtain several independent rectangular images surrounding the target, including: The preprocessed captured image is sent to a trained ice cream detection neural network model with instance segmentation function to output the confidence that each grid contains a target and the relevant information corresponding to the grid, wherein the relevant information includes the target center coordinate information and the corresponding boundary point coordinate information of a preset fixed number P that surrounds the target; Determine in turn whether the confidence level of each grid containing the target is greater than the set threshold; If there is a grid containing a target whose confidence level is greater than the set threshold, the relevant information corresponding to the grid is retained; Determine the polygonal frame of the target based on the target center coordinate information and its corresponding boundary point coordinate information, and then determine the minimum circumscribed rectangular frame of the ice cream based on the polygonal frame; The image within the minimum circumscribed rectangular frame is transformed into a standard image of a preset standard size by using an affine transformation formula; the affine transformation formula includes: Affine transformation matrix The parameters in are specifically solved as follows: by using the length-width ratio of the ice cream structure before and after the affine transformation and its coordinate point information, a superlinear equation group is established to estimate the parameter values ​​of the affine matrix; The target in the polygonal frame area of ​​the standard image is retained, and the pixel values ​​outside the polygonal frame area are uniformly set to preset values ​​to obtain a rectangular image; The ice cream detection neural network model includes: an input end, a reference network, a Neck network and a Head output end; The input end is used to normalize the collected image and scale it to a size of 608*608 before feeding it into the benchmark network; The baseline network is used to extract features. It downsamples layer by layer to obtain feature layers of 76*76, 38*38, and 19*19 sizes, and sends the feature layer of 19*19 size to the Neck network. Neck network, used for connecting and restoring the feature layer of 19*19 size with the feature of the same size on the backbone through upsampling on the channel to obtain feature layers of 38*38 and 76*76 sizes respectively, and sending the obtained feature layers of 19*19, 38*38 and 76*76 sizes to the Head output end, wherein the grid containing the target center coordinates on each feature layer is generated with respect to the x offset and y offset of the upper left corner coordinate of the grid, and at the same time, the x offset and y offset of the boundary points with a fixed number P preset surrounding the target on the feature layer relative to the upper left corner coordinate of the grid are generated, and in addition, the confidence of the target is also generated, with a total of 2+P*2+1 upper limit thresholds, and the feature layer size of each layer output is recorded as S*S, then the feature layer output of each layer is S*S*(2+P*2+1), and the fixed number P is 20; The head output end is used to send the feature layers of 19*19, 38*38 and 76*76 sizes into the convolution kernel to generate the feature image, and then send it to the fully connected layer for full connection, to obtain and output the confidence of each grid containing the target, the target center coordinate information and the corresponding boundary point coordinate information with a fixed number P that surrounds the target; By establishing a multi-class ice cream recognition model to recognize rectangular images, the prediction results are output, the prediction results are fused to obtain the recognition results, and the structured information is sorted out according to the recognition results; Detect whether there is a new collected image, and repeat the above ice cream detection and recognition process for the new collected image.

2. The ice cream detection and identification method for market audit according to claim 1, characterized in that: The ice cream detection neural network model training method comprises: Obtaining a collection image and a sample image of ice cream in a freezer, wherein the sample image has been annotated with various label formats in advance; The collected image is sent to the ice cream detection neural network model to obtain training data with the same label format as the sample image, the training data and the label data are brought into the loss function to calculate the loss, and then the loss is propagated back through the back propagation algorithm to update the parameters of the ice cream detection neural network model. This process is repeated continuously to finally obtain a trained parameter matrix, and the parameter matrix is ​​loaded into the ice cream detection neural network model for updating to complete the training of the ice cream detection neural network model.

3. The ice cream detection and identification method for market audit according to claim 2, characterized in that: The loss function includes: Loss = lobj + loff + lpoly; In the formula, lobj represents the loss of whether there is a target, loff represents the loss of the center point coordinate offset, and lpoly represents the loss of the boundary point coordinate relative to the center point; λ obj ,λ nobj ,λ off ,λ poly They represent the weight coefficients of the loss of obj; S represents the side length of the feature map, and P represents the number of boundary point coordinates; 1 i Indicates whether there is a target in the i-th grid on the nobj feature map. If yes, it is 1, and if not, it is 0; 1 i Indicates whether there is no target in the i-th grid on the feature map, if not, it is 1, if it is, it is 0; c i , They respectively represent the true value and predicted value of whether the i-th grid of the feature map has an object; x i ,y i Respectively represent the actual offset of the target in the ith grid relative to the upper left corner of the grid in the x and y directions, Respectively represent the predicted offset of the target in the ith grid to the upper left corner of the grid in the x and y directions, which are used to determine the actual coordinates of the target center point; smooth l1 represents a regression loss function; x i,j ,y i,j They represent the actual offsets of the coordinates of the P boundary points relative to the upper left corner of the grid in the x and y directions, respectively. They respectively represent the values ​​of the predicted offsets of the coordinates of the P boundary points relative to the upper left corner of the grid in the x and y directions, which are used to determine the bounding box surrounding the target; Loss represents the sum of the loss functions.

4. The ice cream detection and identification method for market audit according to claim 1, characterized in that: The step of preprocessing the collected image comprises: Determine whether the quality of the acquired image is qualified; When the image quality is unsatisfactory, the image is recaptured; When the image quality is qualified, the collected images are uniformly scaled to a size of 608*608, and the collected images are detected and corresponding data enhancement operations are performed, including up and down flipping, Gaussian blur, rotation, simulated occlusion, and simulated glass reflection.

5. An ice cream detection and identification system for market audit, characterized in that: include: An image acquisition module (1) is used to acquire an image of the ice cream in the freezer, pre-process the image, and input the image into the neural network model; The image processing module (2) is used for the ice cream detection neural network model to adopt a neural network model based on yolov5, and the ice cream recognition model to adopt a recognition model based on mobilenet+arcsoftmaxloss; The ice cream detection neural network model extracts, processes and calculates the image information to obtain several independent rectangular images surrounding the target, including: The preprocessed captured image is sent to a trained ice cream detection neural network model with instance segmentation function to output the confidence that each grid contains a target and the relevant information corresponding to the grid, wherein the relevant information includes the target center coordinate information and the corresponding boundary point coordinate information of a preset fixed number P that surrounds the target; Determine in turn whether the confidence level of each grid containing the target is greater than the set threshold; If there is a grid containing a target whose confidence level is greater than the set threshold, the relevant information corresponding to the grid is retained; Determine the polygonal frame of the target based on the target center coordinate information and its corresponding boundary point coordinate information, and then determine the minimum circumscribed rectangular frame of the ice cream based on the polygonal frame; The image within the minimum circumscribed rectangular frame is transformed into a standard image of a preset standard size by using an affine transformation formula; the affine transformation formula includes: Affine transformation matrix The parameters in are specifically solved as follows: by using the length-width ratio of the ice cream structure before and after the affine transformation and its coordinate point information, a superlinear equation group is established to estimate the parameter values ​​of the affine matrix; The target in the polygonal frame area of ​​the standard image is retained, and the pixel values ​​outside the polygonal frame area are uniformly set to preset values ​​to obtain a rectangular image; The ice cream detection neural network model includes: an input end, a reference network, a Neck network and a Head output end; The input end is used to normalize the collected image and scale it to a size of 608*608 before feeding it into the benchmark network; The baseline network is used to extract features. It downsamples layer by layer to obtain feature layers of 76*76, 38*38, and 19*19 sizes, and sends the feature layer of 19*19 size to the Neck network. Neck network, used for connecting and restoring the feature layer of 19*19 size with the feature of the same size on the backbone through upsampling on the channel to obtain feature layers of 38*38 and 76*76 sizes respectively, and sending the obtained feature layers of 19*19, 38*38 and 76*76 sizes to the Head output end, wherein the grid containing the target center coordinates on each feature layer is generated with respect to the x offset and y offset of the upper left corner coordinate of the grid, and at the same time, the x offset and y offset of the boundary points with a fixed number P preset surrounding the target on the feature layer relative to the upper left corner coordinate of the grid are generated, and in addition, the confidence of the target is also generated, with a total of 2+P*2+1 upper limit thresholds, and the feature layer size of each layer output is recorded as S*S, then the feature layer output of each layer is S*S*(2+P*2+1), and the fixed number P is 20; The head output end is used to send the feature layers of 19*19, 38*38 and 76*76 sizes into the convolution kernel to generate the feature image, and then send it to the fully connected layer for full connection, to obtain and output the confidence of each grid containing the target, the target center coordinate information and the corresponding boundary point coordinate information with a fixed number P that surrounds the target; An image recognition module (3) is used to recognize the rectangular image by establishing a multi-class ice cream recognition model, output a prediction result, fuse the prediction result to obtain a recognition result, and sort out structured information based on the recognition result; The re-detection module (4) is used to detect whether a new collected image appears and repeat the above ice cream detection and recognition process for the new collected image.

6. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes the ice cream detection and identification method for market audit as claimed in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The device stores a computer program which can be loaded by a processor and executes the ice cream detection and identification method for market audit as claimed in any one of claims 1 to 4.

Images