Refrigerator inventory detection method and device based on AI vision

Abstract

The invention relates to the technical field of target identification, in particular to a refrigerator inventory detection method and device based on AI vision, in the refrigerator inventory detection method and device based on AI vision, serial numbers of column layer lattice intersection points in a refrigerator image are calibrated through a graph attention network, so that a commodity detection result can be accurately mapped according to structured lattice information, and the detection accuracy is improved. The method comprises the following steps: establishing a two-dimensional detection frame, establishing a connection edge between nodes to reflect a commodity adjacency relationship, realizing conversion from the two-dimensional detection frame to a structured topological graph, effectively improving the stability and robustness of commodity label judgment, and completing deviation verification of a label space position through a Canny edge detection algorithm and in combination with typical column layer commodity distribution frequency, thereby improving the commodity label judgment accuracy. According to the method, the problem of label position mismatching caused by placement deviation or shooting angle change is accurately recognized, the adaptive capacity of refrigerator structure layout is enhanced, the capture precision of commodity visual boundary features is improved, and a commodity recognition result with higher reasoning capacity and structural consistency in a refrigerator scene is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target recognition, in particular to a refrigerator inventory detection method and device based on AI vision. BACKGROUND

[0002] The technical field of target recognition aims to enable computer systems to automatically detect, recognize and locate specific objects from images or videos, thereby accurately extracting information such as object type, quantity, location and posture in images, enabling the system to stably recognize multiple targets in complex and variable visual environments, and widely serving intelligent monitoring, autonomous driving, industrial detection, smart retail and other scenarios.

[0003] The purpose of the refrigerator inventory detection method based on AI vision is to realize automatic detection and updating of refrigerator inventory, eliminate the labor cost and error risk of manual inventory, and improve retail operation efficiency and replenishment response speed, aiming to ensure real-time and accurate acquisition of refrigerator product information, thereby optimizing product display, replenishment cycle and inventory management strategy, and achieving unmanned and intelligent operation goals.

[0004] The existing technology directly outputs product location and type information based on a two-dimensional image target detection and recognition model, without considering the constraints and logical relationships of the refrigerator grid structure on the recognition label, resulting in label drift and mislabeling problems in scenarios such as misaligned product placement, overlapping and shielding, and critical confidence. Since it does not have a graph structure relationship modeling mechanism, when adjacent products have similar categories or detection frame boundaries intersect, it cannot effectively correct the label through context information, making it difficult to accurately identify spatially misplaced products caused by shooting angles or display methods, causing label mapping position deviation and affecting the consistency of structured output. SUMMARY

[0005] The purpose of the present application is to solve the shortcomings in the prior art, and to provide a refrigerator inventory detection method and device based on AI vision.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme: a refrigerator inventory detection method based on AI vision, comprising the following steps: S1: based on the column layer number index in the set refrigerator grid structure, using a graph attention network, calibrating and numbering the node index of each layer and each column intersection in the refrigerator front view image, extracting the corresponding grid of the product detection frame in the refrigerator front view image, establishing node connection edges by judging the co-occurrence frequency and actual adjacency relationship of adjacent grid products, and generating a node graph connection relationship table; S2: based on the node graph connection relationship table, reading the product type and confidence of each grid, filtering and judging the product label, and if the label is inconsistent with the adjacent node label, uniform the label, and generate a label consistency correction node set; S3: Based on the label consistency correction node set, adopting a Canny edge detection algorithm, the distribution frequency of the typical column layer of the commodity category in the refrigerator is counted, the difference between the current node position and the peak position of the typical column layer distribution frequency is calculated, and the corner profile difference of the grid image cut out according to the node coordinate area is compared, the node with inconsistent spatial offset and visual information is identified, and a position visual conflict node list is generated; S4: Based on the position visual conflict node list, combined with the commodity main category and subcategory table, the coincidence relationship of the candidate label and the adjacent label on the category path is analyzed, the path offset is judged, the replaceable label is determined, the hierarchical reasoning updated label set is obtained; S5: Based on the hierarchical reasoning updated label set, the node commodity label is rearranged according to the column layer sequence, and is mapped to the corresponding position of the image, and the structured panoramic recognition result is output, and a structure mapping recognition label matrix is generated.

[0007] As a further scheme of the present application, the specific steps for generating the node graph connection relationship table are: Based on the column layer number index in the set refrigerator grid structure, the order of all column line and layer line intersection coordinates is extracted and matrix number processing is performed, and the intersection coordinate position is converted into a two-dimensional index table structure, a unique number set of grid nodes is established, and a grid node coordinate index set is generated; Based on the grid node coordinate index set, the upper left corner and lower right corner coordinates of the commodity detection box are extracted, and the inclusion relationship between the four corner coordinates of the region and the node index region boundary is judged, the hit node is determined and the commodity label and category number are bound, and a grid commodity binding record table is generated; Based on the grid commodity binding record table, the column layer value difference of all node indexes is calculated by using the graph attention network, the commodity category label pairs corresponding to adjacent node indexes are extracted, the appearance frequency of each group of label pairs is counted, the statistical frequency is compared with the set threshold value, the node pairs meeting the conditions are screened, the edge connection information is written, and the node graph connection relationship table is generated.

[0008] As a further scheme of the present application, the specific steps for generating the label consistency correction node set are: Based on the node graph connection relationship table, all grid node commodity category labels are read and confidence value matching is performed, node indexes with confidence values lower than the threshold value are screened, and a set of these node numbers is recorded, a low-confidence node index set is generated, and a commodity label distribution table is established to record the current commodity category label state of each node in the grid node coordinate index set; Based on the low-confidence node index set, the connection edge index of each node is retrieved, the commodity labels in the connected nodes are extracted and the frequency values are counted according to the category, and then the consistency of the label with the highest frequency and the current label is judged to determine the replacement node content, and a to-be-replaced node label mapping table is generated. Based on the to-be-replaced node label mapping table, the current node original label is replaced by the highest frequency label, and the node new label content is written into the grid node coordinate index set, the commodity label distribution table content is updated, and the label consistency correction node set is obtained.

[0009] As a further scheme of the present application, the specific steps for generating the position visual conflict node list are: Based on the label consistency correction node set, the frequency statistics of the node label under the column layer number index are carried out, all commodity category labels are grouped according to the column layer number index, the number of labels in each group is counted, and the column layer position corresponding to the maximum value is extracted, and a label frequency peak index table is generated; Based on the label frequency peak index table, the difference between the current node column layer index and the frequency peak index is calculated, and the difference is compared with the set threshold value item by item, the nodes with the difference greater than the threshold value are screened and the node number is extracted, and a position offset node index set is generated; Based on the position offset node index set, the Canny edge detection algorithm is used to extract the edge corner profile of the four boundary regions in the node corresponding image region and convert it into a binary graph structure, the difference intensity of the commodity standard sample graph in the database is compared by calculating the profile area, boundary number and gray level variation degree, and a position visual conflict node list is generated.

[0010] As a further scheme of the present application, the specific steps for generating the hierarchical reasoning updated label set are: Based on the position visual conflict node list, the current label and the candidate label of each node are extracted, the main class path and the sub-class index of each label are retrieved by table lookup method, and the path node number list is established, and a candidate label path set table is generated; Based on the candidate label path set table, the number intersection between the candidate path node set and the adjacent node path set is obtained, and the intersection node number and the path offset distance are counted, and the qualified labels are screened by comparing the set threshold value, and a path corresponding replaceable label table is generated; Based on the path corresponding replaceable label table, the current node label content in the label consistency correction node set is replaced by the path coincident and offset compliant candidate label, and the replacement result is written into the grid node coordinate index set corresponding node position, the whole node label set is updated, and the hierarchical reasoning updated label set is obtained.

[0011] As a further scheme of the present application, the specific steps for generating the structure mapping identification label matrix are: Based on the hierarchical reasoning, the label set is updated, column number extraction in node index is performed, and ascending order sorting is executed, layer number extraction is performed after column number classification, and ascending order arrangement is executed, the node index is sorted in ascending order of column number and arranged in ascending order of layer number through a double-layer nested loop structure, and a column-layer rearranged node index table is generated; Based on the column-layer rearranged node index table, the corresponding commodity label of each node is extracted, and the node coordinate area range is read, the region top-left and bottom-right coordinates are converted into image indexes, the region is calibrated, the commodity label is written into the corresponding pixel matrix position, and a node label image mapping table is generated; Based on the node label image mapping table, each calibrated image region is extracted, and row and column filling operations of the image matrix are performed, the same column and different layer annotation regions are sequentially superimposed, and the whole image splicing structure is completed, and a structure mapping recognition label matrix is generated.

[0012] An AI vision-based refrigerator inventory detection device, comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps of the AI vision-based refrigerator inventory detection method.

[0013] Compared with the prior art, the advantages and positive effects of the present application are that: In the present application, the numbering of the column-layer grid intersection points in the refrigerator image is calibrated by the graph attention network, so that the commodity detection result can be accurately mapped according to the structured grid information, and the connection edges between nodes are established to reflect the adjacency relationship of the commodities, realizing the conversion from the two-dimensional detection frame to the structured topological graph, effectively improving the stability and robustness of the commodity label determination, and significantly alleviating the misidentification and misattribution problems caused by the occlusion between commodities, the inclination of placement or the lens shift; In the present application, the deviation of the label space position is verified by the Canny edge detection algorithm combined with the typical column-layer commodity distribution frequency, the label position mismatch problem caused by the placement shift or the change of the shooting angle is accurately identified, the adaptation ability to the refrigerator structure layout is enhanced, the capture precision of the commodity visual boundary feature is improved, and the commodity recognition result with higher reasoning ability and structural consistency for the refrigerator scene is realized. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The present application is a workflow schematic diagram. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application.

[0016] In the description of the present invention, it should be understood that the terms "length," "width," "up," "down," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inside," "outside," and the like, indicating positions or relationships, are based on the positions or relationships shown in the accompanying drawings and are intended only to facilitate the description of the present invention and simplify the description. They do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, in the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined.

[0017] Example 1 See also Figure 1 The present invention provides a technical solution: a freezer inventory detection method based on AI vision, comprising the following steps: S1: Based on the column and layer number index in the preset freezer grid structure, a graph attention network is used to calibrate the coordinates of the intersection points of each layer and column in the freezer front view image and number the node indexes. The corresponding grid positions of the product detection frames in the freezer front view image are extracted. By determining the co-occurrence frequency of products in adjacent grids and the actual adjacency relationship, node connection edges are established to generate a node graph connection relationship table. S2: Based on the node graph connection relationship table, read the product type and confidence of each grid, filter and determine the product label. If the label is inconsistent with the adjacent node label, unify the label and generate a label consistency correction node set; S3: Based on label consistency, the node set is corrected and the Canny edge detection algorithm is used to count the frequency of typical column-level distribution of product types in the refrigerator. The difference between the current node position and the peak position of the typical column-level distribution frequency is calculated. At the same time, the grid image cropped according to the node coordinate area is compared for corner contour differences to identify nodes whose spatial offset does not match the visual information, and a list of nodes with positional visual conflicts is generated. S4: Based on the list of positional visual conflict nodes and combined with the product main category and subcategory tables, the overlap relationship between candidate labels and adjacent labels on the category path is analyzed, the path offset is determined, the replaceable labels are determined, and the hierarchical reasoning update label set is obtained; S5: Update the label set based on hierarchical reasoning, rearrange the node product labels in column-level order, and map them to the corresponding positions in the image. Output the structured panoramic recognition results and generate a structured mapping recognition label matrix.

[0018] The specific steps to generate the node graph connection relationship table are: Based on the set column and layer number index in the refrigerator grid structure, the coordinates of the intersections of all column lines and layer lines are sequentially extracted and matrix numbered, and the intersection coordinate positions are converted into a two-dimensional index table structure. A unique number set of grid nodes is established, and a grid node coordinate index set is generated. Based on the grid node coordinate index set, the upper left corner and the lower right corner coordinates of the commodity detection frame are extracted, and the inclusion relationship between the four corner coordinates of the region and the node index region boundary is judged to determine the hit node and bind the commodity label and the category number, and a grid commodity binding record table is generated; Based on the grid commodity binding record table, the column layer value difference of all node indexes is calculated using a graph attention network, the commodity category label pairs corresponding to adjacent node indexes are extracted, the appearance frequency of each label pair is counted, the statistical frequency is compared with the set threshold, the node pairs that meet the conditions are screened, the connection information is written, and a node graph connection relationship table is generated. Based on the column layer number index in the set refrigerator grid structure, a two-dimensional coordinate mapping method is used to sample and process the intersection points of column lines and layer lines, the number of column lines and the number of layer lines are set to correspond to the actual physical structure of the refrigerator in sequence, the pixel coordinate values of all intersection points are extracted in the order of appearance in the image, and a unique two-dimensional index number is established for each intersection point. The two-dimensional index table structure is generated by sequentially assigning values; the key value mapping of each intersection point number and the corresponding coordinate position is performed to construct a one-to-one mapping set from node number to coordinate value, record all grid node numbers and coordinate information, and generate a grid node coordinate index set. Based on the grid node coordinate index set, the positions of the upper left corner and the lower right corner of the commodity detection frame are extracted, the four corner coordinate boundaries of each detection frame are confirmed by sequential traversal, the inclusion relationship of all node coordinates in the node index set is judged, and the condition is whether the node is in the boundary rectangle formed by the upper left corner and the lower right corner of the detection frame. If it is satisfied, the node number is marked as a hit node; the corresponding commodity label and category number are written for each hit node, and the node number and the commodity information are recorded as a key-value mapping table structure, and the results are summarized after performing the node binding operation item by item to generate a grid commodity binding record table. Based on the grid commodity binding record table, the graph attention network algorithm is used to calculate the connection relationship between nodes, the column number and the layer number of all nodes are extracted, the column layer number difference is calculated for each pair of nodes to identify whether they are adjacent grids in the physical structure; the commodity category number of the bound nodes is extracted and combined into a commodity category combination; the statistical frequency of all commodity combinations is calculated and compared with the preset threshold to screen the node pairs whose frequency meets the threshold requirement; the selected node pairs are written into the connection relationship information, the start and target numbers of the node connection are recorded, and the connection edge data structure is constructed to generate a node graph connection relationship table.

[0019] The graph attention network is according to the formula: Wherein: h′ idenotes the representation vector of node i after the graph neural network embedding update, denotes the set of all adjacent nodes that have a coincident relationship with the combination of product categories of node i in the refrigerator layout, a ij denotes the basic attention weight calculated between node i and adjacent node j through the graph attention mechanism, b ij denotes the combination label coincidence degree of the set of product categories between node i and node j, g ij denotes the difference suppression factor between node i and node j in the index column layer dimension, W denotes the weight matrix for linear transformation of the input node feature vector in the graph neural network, h j denotes the original input feature vector of adjacent node j, and s denotes a nonlinear activation function. The execution process is as follows: first, the system extracts the basic information of each grid node in the refrigerator through the visual recognition module, constructs the input feature vector h j of each node, including the current grid binding product category code, the number of product combinations, the grid number, etc., which is mapped to a unified embedding space via W, and then analyzes the product binding record table to identify node pairs with product combination relationships, and establishes the For each pair of nodes (i, j), first calculate the distance suppression factor g ij according to the column layer position difference of the node index, then extract the intersection-union ratio of the product set of node i and node j, and calculate the combination label coincidence degree b ij to measure the similarity of the two nodes in terms of product categories, then concatenate the feature vectors, calculate the basic attention weight a ij through the single-hidden-layer attention network, use LeakyReLU as the activation function and normalize the processing, and finally multiply all the embedding vectors Wh j of adjacent node j by a ij , b ij , and g ij , and then weightedly sum them up, and input the converged result into the nonlinear activation function s to obtain the updated representation vector of node i, complete the semantic association modeling between the grid nodes in the refrigerator graph structure, and generate the subsequent inventory state reasoning and product replenishment suggestion.

[0020] The specific steps for generating the label consistency correction node set are as follows: Based on the node graph connection relationship table, read the product category labels of all grid nodes and match the confidence values, filter the node indexes with confidence values below the threshold, record these node numbers to construct a set, generate a low-confidence node index set, and at the same time establish a product label distribution table to record the current product category label state of each node in the grid node coordinate index set. Based on the low confidence node index set, the connection edge index of each node is retrieved, the commodity label in the connected node is extracted, and the frequency value is counted according to the type, and then the label with the highest frequency is compared with the current label for consistency, the node content is replaced, and a label mapping table of the node to be replaced is generated; Based on the label mapping table of the node to be replaced, the original label of the current node is replaced by the label with the highest frequency, and the new label content of the node is written into the grid node coordinate index set, the content of the commodity label distribution table is updated, and a set of nodes with consistent labels is obtained. Based on the node graph connection relationship table, a confidence threshold screening method is used to match all grid node commodity type labels. First, read the commodity type label bound to each node and the corresponding confidence value. Set the confidence threshold to 0.6. Compare the confidence value of each node with the threshold. Use a sequential judgment structure to perform confidence comparison in order of node number. If the confidence is less than the threshold, record the node number in the temporary storage list. After traversing all nodes, integrate all node numbers with substandard confidence, and perform de-duplication and sorting processing on the number list to construct a unified number set structure, and generate a low confidence node index set. Based on the low confidence node index set, a node connection index retrieval method is used to extract and process the connection edge set of each low confidence node. For each node number in the set, retrieve all target node numbers in its connection relationship one by one, read the commodity label bound to the target node, and count the frequency value according to the label type by grouping. Sort the statistical results according to the label frequency, extract the label item with the highest frequency as the candidate label, and compare it with the original label of the current node. If the label values are inconsistent, record the current node number and the candidate label as a key-value pair. Collect all node numbers to be replaced and the corresponding label content to generate a label mapping table of the node to be replaced. Based on the label mapping table of the node to be replaced, a label updating method is used to modify the label information of all nodes to be replaced. Access each record in the mapping table in order of number, perform the original label covering operation on the current node, replace the label with the highest frequency directly, and write the replacement operation to the label update index set according to the node number. At the same time, update the label table structure, including covering the original label field value, updating the label version number field and modifying the time stamp record field. Write all changed information to the unified label update index table. Finally, output the node number and label information set after updating all label contents, and generate a set of nodes with consistent labels.

[0021] The specific steps for generating the position visual conflict node list are as follows: Based on the label consistency correction node set, the frequency statistics of the node label under the column layer number index are performed, all commodity category labels are grouped according to the column layer number index, the number of labels in each group is counted, and the column layer position corresponding to the maximum value is extracted to generate a label frequency peak index table; Based on the label frequency peak index table, the difference between the current node column layer index and the frequency peak index is calculated, and the difference is compared with the set threshold value item by item, the nodes with a difference greater than the threshold value are screened and the node number is extracted to generate a position offset node index set; Based on the position offset node index set, the Canny edge detection algorithm is used to extract the edge corner profile of the four boundary regions in the node corresponding image region and convert it into a binary graph structure, the difference intensity of the commodity standard sample graph in the database is compared by calculating the profile area, boundary number and gray variation degree, and a position visual conflict node list is generated. Based on the label consistency correction node set, the frequency statistics of the node label under the column layer number index are performed, all commodity category labels are grouped according to the column layer number index, the number of labels in each group is counted, and the column layer position corresponding to the maximum value is extracted to generate a label frequency peak index table; Based on the label frequency peak index table, the difference between the current node column layer index and the frequency peak index is calculated, and the difference is compared with the set threshold value item by item, the nodes with a difference greater than the threshold value are screened and the node number is extracted to generate a position offset node index set; Based on the position offset node index set, the Canny edge detection algorithm is used to perform edge corner profile feature extraction on the image area corresponding to the node. First, read the image segment area corresponding to each node and perform gray scale conversion processing. Set the high threshold value to 150 and the low threshold value to 50. Use the Canny algorithm to perform edge recognition on the converted image to generate an edge profile binary graph. Then perform profile area statistics on the binary graph to calculate the total length of the edge profile and the number of closed boundary areas. At the same time, calculate the variation degree value of the image gray scale, collect the regional standard deviation and the average value of the pixel gradient distribution, and use the absolute difference method to compare the numerical difference in three dimensions with the standard product image. Execute the difference judgment item by item and record the node number and difference value that is higher than the set deviation threshold. Finally, all the judgment results are combined and recorded as a number and conflict index key-value pair structure to generate a position visual conflict node list.

[0022] The Canny edge detection algorithm is as follows: Wherein: represents the edge intensity value after direction, gray scale and structure difference weighting, represents the horizontal direction coordinate axis of the image, represents the vertical direction coordinate axis of the image, represents the gradient value of each pixel point in the horizontal direction of the image, represents the gradient value of each pixel point in the vertical direction of the image, represents the weighting coefficient of the horizontal direction gradient component, represents the weighting coefficient of the vertical direction gradient component, represents the standard deviation of the gray scale value in the image area, represents the gray scale variation adjustment coefficient, represents the similarity between the image area and the standard product sample image profile structure, represents the structure difference adjustment coefficient; The execution process is as follows: first, extract the corresponding image area and perform Gaussian filtering to reduce the noise influence. Then use the Sobel operator to calculate the gradient value of each pixel point in the horizontal direction and the vertical direction of the image, respectively represented as and To enhance the discrimination accuracy of the actual refrigerator product edge shape, the direction adjustment weight and are introduced, which respectively represent the control degree of the system to the transverse edge definition and the longitudinal edge noise resistance, and the gray scale standard deviation of the region is calculated to reflect the local texture complexity, and the influence on the final edge response is adjusted through the empirical coefficient Then extract the corresponding area profile from the standard product sample image and calculate the structure similarity through shape matching , and the structural difference adjustment coefficient The influence intensity is regulated, and all the parameters are substituted into the improved formula to calculate the enhanced edge intensity value Finally, the edge corner contour is extracted in the formed edge binary graph, and the area, boundary number and gray scale variation characteristics are counted and compared with the standard template graph. If the difference exceeds the set threshold, it is marked as an abnormal node with occlusion, mixed placement or image ghosting, and added to the position visual conflict node list.

[0023] The specific steps of generating the hierarchical reasoning updated label set are: Based on the position visual conflict node list, the current label and the candidate label of each node are extracted, the main class path and the sub-class index of each label are retrieved through table lookup, and the path node number list is established to generate the candidate label path set table; Based on the candidate label path set table, the number intersection of the candidate path node set and the adjacent node path set is obtained, and the intersection node number and the path offset distance are counted. The qualified labels are screened by comparing the set threshold to generate the path corresponding replaceable label table; Based on the path corresponding replaceable label table, the current node label content in the label consistency correction node set is replaced by the path coinciding and offset compliant candidate label, and the replacement result is written into the node position corresponding to the grid node coordinate index set. Update the entire node label set to obtain the hierarchical reasoning updated label set; Based on the position visual conflict node list, the main class path and the sub-class index of the current label and the candidate label of each node are extracted by using the path table lookup matching method. First, read the node binding label and the candidate label list. After each label is converted into a number, it is used as a query key value. The main class path node number sequence and the sub-class index number corresponding to the label are retrieved one by one in the class path lookup table. The path node number is stored as a one-dimensional path number array in the form of a sequential list. The node path number record of the current label and the candidate label is established, and after the path number is archived, the node number and the corresponding label path number are processed by key value mapping. The label path content is integrated and structured to form a structured label path set, and the candidate label path set table is generated; Based on the candidate label path set table, the intersection matching and path distance calculation method is used to compare the intersection of the candidate path and the adjacent node path set. The number of each candidate label path set is compared with the number of its adjacent node label path set, the intersection number between the two sets is obtained, and the number of intersection nodes is counted as the basis for label similarity judgment. At the same time, the absolute difference value of the path node position number is calculated, and the path offset distance between the candidate label path number set and the adjacent label path set is counted. The intersection node number and the offset distance are compared with the preset threshold value item by item, and the threshold value is set to be not less than 2 intersection numbers and not more than 3 path offsets. The label path number and the corresponding candidate label that meet the threshold requirement are recorded in a structured manner to generate a path corresponding replaceable label table; Based on the path corresponding replaceable label table, the label value replacement operation is used to modify the original node label content. For each candidate label item recorded in the mapping table, the label value replacement operation is performed on the original node number. The replacement result is written into the original node index structure, and the node label mapping set is updated synchronously. The time mark field and the replacement state field are added to the processed node number to record the processing behavior. The node label set is renumbered and indexed to integrate all updated node numbers and the latest label content, and a hierarchical reasoning update label set is generated.

[0024] The specific steps for generating the structure mapping recognition label matrix are as follows: Based on the hierarchical reasoning update label set, the column number in the node index is extracted and sorted in ascending order. The layer number is extracted and sorted in ascending order after classification by column number. The node index is sorted in ascending order by column number and in ascending order by layer number through double nested loop structure, and a column-layer rearranged node index table is generated. Based on the column-layer rearranged node index table, the corresponding product label of each node is extracted and the node coordinate area range is read. The left upper corner and the right lower corner coordinates of the area are converted into image indexes and calibrated, and then the product label is written into the corresponding pixel matrix position to generate a node label image mapping table. Based on the node label image mapping table, each calibrated image area is extracted and the row and column filling operation of the image matrix is performed. The same column and different layer annotation areas are sequentially superimposed and the whole image splicing structure is completed to generate a structure mapping recognition label matrix. The label set is updated based on hierarchical reasoning, and the node index position sequence is arranged using a nested sorting structure. First, the column number field is extracted from all node index structures and an ascending sorting operation is performed. The sorting is performed in a stable sorting manner, and after arranging from small to large according to the column number, the layer number field of each column is extracted, and a local ascending sorting operation is performed. After sorting, the outer column number set and the inner layer number set are traversed using a double-layer nested loop structure, a new continuous index value is assigned to each column layer combination, and a mapping relationship between the new and old index numbers is established. The mapping results between the new and old index numbers are recorded as key-value structures. After integrating all node rearrangement number structures, a column layer rearrangement node index table is generated. Based on the column layer rearrangement node index table, the node label and the region position are annotated using an image coordinate mapping method. First, the left upper corner and the right lower corner coordinates of the node region in the original image corresponding to the commodity label content of each node are extracted, and the pixel coordinate conversion processing is performed on the coordinate pair. The conversion method is to map the coordinates (x, y) to the two-dimensional index position in the image matrix based on rows and columns. The access and marking operations are performed on all pixel points in the rectangular region using the row-by-row and column-by-column mapping logic. In the access process, the commodity label information is written into each pixel corresponding matrix unit, and the binding relationship between the pixel calibration boundary and the label value is recorded. After the annotation operation is performed on all node regions, the image pixel marking matrix is unified and merged to generate a node label image mapping table. Based on the node label image mapping table, the matrix level splicing processing is performed on the calibrated image region using the column-by-column splicing operation. First, the image matrix region corresponding to all layer numbers is extracted for each column number, and the vertical superposition operation is performed from top to bottom according to the layer number. After aligning the matrix corresponding to each layer according to the column direction position, the matrix padding operation is used to fill the height difference of different regions. The padding method is set to a constant padding value and the label value is kept unchanged. After completing the splicing of all image blocks under the same column, the horizontal splicing processing is performed on all column image structures in the column order. The different column image regions are spliced in the left-to-right order and a new image matrix index is established. The row and column coordinate mapping table and the label index table are synchronously recorded for the integrated image matrix structure to generate a structure mapping recognition label matrix.

[0025] An AI vision-based refrigerator inventory detection device, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the AI vision-based refrigerator inventory detection method.

[0026] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in other forms. Any skilled person in the art can modify or change the disclosed technical content into equivalent embodiments with equivalent changes, and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the technical solution content of the present application, still falls within the protection scope of the present application.

Claims

1. A refrigerator inventory detection method based on AI vision, characterized in that: The following steps are involved: S1: Based on the column and layer number index in the preset freezer grid structure, a graph attention network is used to calibrate the coordinates of the intersection points of each layer and column in the freezer front view image and number the node indexes. The corresponding grid positions of the product detection frames in the freezer front view image are extracted. By determining the co-occurrence frequency of products in adjacent grids and the actual adjacency relationship, node connection edges are established to generate a node graph connection relationship table. S2: Based on the node graph connection relationship table, read the product category and confidence level of each grid, determine the product label after screening, and generate a label consistency correction node set; S3: Based on the label consistency correction node set, the Canny edge detection algorithm is used to count the typical column-layer distribution frequencies of commodity types in the refrigerator, and the difference between the current node position and the peak position of the typical column-layer distribution frequency is calculated. At the same time, the grid image cropped according to the node coordinate area is compared for edge and corner contour differences to identify nodes whose spatial offsets do not match the visual information, and generate a list of nodes with positional visual conflicts; S4: Based on the position visual conflict node list, extract the labels under the node main category path as candidate labels, analyze the overlap relationship between the candidate labels and the adjacent node labels on the category path, determine the path offset, determine the replaceable labels, and obtain the hierarchical reasoning update label set; S5: Based on the hierarchical reasoning, the label set is updated, the node product labels are rearranged in column-level order, and mapped to the corresponding positions of the image, the structured panoramic recognition results are output, and a structure mapping recognition label matrix is ​​generated.

2. The AI ​​vision-based freezer inventory detection method according to claim 1 is characterized in that: The specific steps of generating the node graph connection relationship table are: Based on the set column and layer number index in the refrigerator grid structure, the coordinates of the intersections of all column lines and layer lines are sequentially extracted and matrix numbered, and the intersection coordinate positions are converted into a two-dimensional index table structure. A unique number set of grid nodes is established, and a grid node coordinate index set is generated. Based on the grid node coordinate index set, the coordinates of the upper left and lower right corners of the product detection frame are extracted, and the inclusion relationship between the coordinates of the four corners of the area and the boundary of the node index area is determined. The hit node is determined and the product label and category number are bound to generate a grid product binding record table; Based on the grid commodity binding record table, a graph attention network is used to calculate the column-level value differences of all node indexes, extract the commodity category label pairs corresponding to adjacent node indexes, count the occurrence frequency of each group of label pairs, compare the statistical frequency with the set threshold, screen the node pairs that meet the conditions, write the edge connection information, and generate a node graph connection relationship table.

3. The AI ​​vision-based freezer inventory detection method according to claim 1, characterized in that: The specific steps of generating the label consistency correction node set are: Based on the node graph connection relationship table, all grid node commodity category labels are read and matched with confidence values, and node indexes with confidence values ​​lower than the threshold are screened. These node numbers are recorded to build a set, generating a low-confidence node index set. At the same time, a commodity label distribution table is established to record the current commodity category label status corresponding to each node in the grid node coordinate index set; Based on the low-confidence node index set, perform a connection edge index search for each node, extract the product labels in the connection nodes and count the frequency values ​​by type, then perform a consistency check between the label with the highest frequency and the current label, determine the replacement node content, and generate a label mapping table for the node to be replaced; Based on the node label mapping table to be replaced, the original label of the current node is replaced with the label with the highest frequency, and the new label content of the node is written to the grid node coordinate index set, the content of the product label distribution table is updated, and the label consistency correction node set is obtained.

4. The AI ​​vision-based freezer inventory detection method according to claim 1, characterized in that: The specific steps of generating the position visual conflict node list are as follows: Based on the label consistency correction node set, the frequency statistics of the node labels under the column layer number index are performed. By grouping all product category labels according to the column layer number index, counting the number of labels in each group and extracting the column layer position corresponding to the maximum value, a label frequency peak index table is generated; Based on the label frequency peak index table, the difference between the current node column layer index and the frequency peak index is calculated, and the difference is compared with the set threshold item by item, and the nodes with the difference greater than the threshold are screened and the node numbers are extracted to generate a position offset node index set; Based on the position offset node index set, the Canny edge detection algorithm is used to extract the corner contours of the four boundary areas in the node corresponding image area and convert them into a binary graph structure. By calculating the contour area, the number of boundaries and the degree of grayscale variation, and comparing the difference intensity of the standard sample images of the goods in the database, a list of position visual conflict nodes is generated.

5. The AI ​​vision-based freezer inventory detection method according to claim 1, characterized in that: The specific steps for generating the hierarchical reasoning update label set are: Based on the position visual conflict node list, extract the current label and candidate label of each node, retrieve the main class path and subclass index of each label by table lookup, establish a path node number list, and generate a candidate label path set table; Based on the candidate label path set table, obtain the number intersection between the candidate path node set and the adjacent node path set, count the number of intersection nodes and the path offset distance, compare with the set threshold to screen qualified labels, and generate a path corresponding replaceable label table; Based on the path corresponding replaceable label table, the current node label content in the label consistency correction node set is replaced with a candidate label with overlapping paths and compliant offsets, and the replacement result is written into the corresponding node position of the grid node coordinate index set, and the entire node label set is updated to obtain the hierarchical reasoning updated label set.

6. The AI ​​vision-based freezer inventory detection method according to claim 1, characterized in that: The specific steps of generating the structure mapping identification label matrix are: Based on the hierarchical reasoning, the label set is updated, the column numbers in the node index are extracted and sorted in ascending order, the layer numbers are extracted after classification by column number and arranged in ascending order, and the node index is sorted in ascending order by column number and in ascending order by layer number through a double-layer nested loop structure to generate a column-layer rearranged node index table; Based on the column-level reordered node index table, the product label corresponding to each node is extracted and the node coordinate area range is read. The coordinates of the upper left corner and lower right corner of the area are converted into image indexes and the area is marked. The product label is then written into the corresponding pixel matrix position to generate a node label image mapping table. Based on the node label image mapping table, each calibrated image area is extracted and the row and column filling operation of the image matrix is ​​performed. By sequentially superimposing the labeled areas of different layers in the same column and completing the whole image splicing structure, a structure mapping recognition label matrix is ​​generated.

7. A refrigerator inventory detection device based on AI vision, comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the processor executes the computer program, the steps of the AI ​​vision-based refrigerator inventory detection method according to any one of claims 1 to 7 are implemented.

Images