Method and system for extracting material pile morphology characteristics

Abstract

The application discloses a kind of material accumulation form feature extraction method and system, the present application relates to the technical field of feature extraction method, according to the matching of the accumulation event of electronic equipment material and the use schedule of each electronic equipment material Determine the theoretical accumulation form of electronic equipment material;Determine the actual accumulation form of electronic equipment material based on the accumulation identification of real-time storage image, according to the actual accumulation form and the theoretical accumulation form of electronic equipment material Determine the abnormal accumulation form feature of electronic equipment material, improve the accuracy of the abnormal accumulation form feature of electronic equipment material.According to the use order of a plurality of abnormal accumulation positions and corresponding electronic equipment material Determine the region to be optimized;Based on the position of the region to be optimized and the accumulation form of the surrounding electronic equipment material Response adjacent material robot, trigger the material robot to adjust the position of electronic equipment material in the region to be optimized, to realize the rationalization of electronic equipment material accumulation.

Description

Technical Field

[0001] This invention relates to the technical field of feature extraction methods, and in particular to a method and system for extracting material accumulation morphology features. Background Technology

[0002] With the development of technology, electronic equipment materials are piled up in corresponding storage areas. There are multiple electronic equipment materials, which are placed at different heights in the same location. These multiple electronic equipment materials are generally piled up in the corresponding storage area. The piles of multiple electronic equipment materials are usually arranged manually according to the current pile situation. However, the piles of multiple electronic equipment materials are sometimes easy to become chaotic, resulting in abnormal pile formations during the pile-up process, making it impossible to achieve rational pile-up of electronic equipment materials. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a method and system for extracting material accumulation morphology features.

[0004] This invention provides a method for extracting material stacking morphology features, comprising: acquiring real-time stored images of electronic device materials; determining the relative positions and shapes of multiple electronic device materials based on material recognition from the real-time stored images; marking contact positions between multiple electronic device materials; determining stacking events of electronic device materials based on multiple contact positions, relative positions, and shapes of the multiple electronic device materials; determining the theoretical stacking morphology of electronic device materials based on the matching of stacking events and usage schedules of each electronic device material; determining the actual stacking morphology of electronic device materials based on stacking recognition from the real-time stored images; determining abnormal stacking morphology features of electronic device materials based on the actual stacking morphology and theoretical stacking morphology; marking multiple abnormal stacking locations of abnormal stacking morphology features; and determining the area to be optimized based on the multiple abnormal stacking locations and the corresponding usage order of electronic device materials.

[0005] Based on the location of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials, the system responds to the nearby material robot and triggers the material robot to adjust the position of the electronic equipment materials in the area to be optimized.

[0006] This invention provides a material accumulation morphology feature extraction system, which is applied to the above-described material accumulation morphology feature extraction method. The material accumulation morphology feature extraction system includes:

[0007] The material identification module is used to acquire real-time stored images of materials in electronic devices, and to determine the relative positions and shapes of multiple electronic device materials based on the material identification of these real-time stored images.

[0008] The theoretical stacking pattern module is used to mark the contact positions between multiple electronic device materials, and to determine the stacking events of the electronic device materials based on the multiple contact positions, the relative positions of the multiple electronic device materials, and their own shapes; and to determine the theoretical stacking pattern of the electronic device materials based on the matching of the stacking events of the electronic device materials and the usage schedule of each electronic device material.

[0009] The abnormal stacking morphology feature module is used to determine the actual stacking morphology of electronic equipment materials based on stacking recognition of real-time stored images, and to determine the abnormal stacking morphology features of electronic equipment materials based on the actual stacking morphology and theoretical stacking morphology of electronic equipment materials.

[0010] The "Region to be Optimized" module is used to mark multiple abnormal stacking locations with abnormal stacking morphology characteristics, and to determine the region to be optimized based on the multiple abnormal stacking locations and the usage order of the corresponding electronic equipment materials.

[0011] The position adjustment module is used to respond to the nearby material robot based on the position of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials, and to trigger the material robot to adjust the position of the electronic equipment materials in the area to be optimized.

[0012] Compared with the prior art, the beneficial effects of the present invention are:

[0013] In this embodiment of the invention, the theoretical stacking pattern of electronic equipment materials is determined by matching the stacking events of electronic equipment materials with the usage schedules of each electronic equipment material; the actual stacking pattern of electronic equipment materials is determined based on the stacking recognition of real-time stored images; and the abnormal stacking pattern features of electronic equipment materials are determined based on the actual stacking pattern and the theoretical stacking pattern. The introduction of the theoretical stacking pattern enables the comparison between the actual stacking pattern and the theoretical stacking pattern of electronic equipment materials, realizes the feature extraction of abnormal stacking pattern features of electronic equipment materials, and improves the accuracy of abnormal stacking pattern features of electronic equipment materials.

[0014] Therefore, multiple abnormal stacking locations with abnormal stacking morphology characteristics are marked, and the area to be optimized is determined based on the multiple abnormal stacking locations and the usage order of the corresponding electronic equipment materials. Based on the location of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials, the system responds to the nearby material robots and triggers the material robots to adjust the position of the electronic equipment materials in the area to be optimized. This introduces the area to be optimized, realizes further control of the area to be optimized, and then responds to the nearby material robots. Through the cooperation of the material robots, the position of the electronic equipment materials in the area to be optimized is adjusted to achieve rational stacking of electronic equipment materials. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the material accumulation morphology feature extraction method in an embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram of the structural composition of the material accumulation morphology feature extraction system in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0018] Please see Figure 1 and Figure 2 A method for extracting material accumulation morphology features, applied to scenarios involving the extraction of material accumulation morphology features; the method includes:

[0019] Step S11: Acquire real-time stored images of materials in electronic devices, and determine the relative positions and shapes of multiple electronic device materials based on material recognition of the real-time stored images;

[0020] Step S12: Mark the contact positions between multiple electronic device materials, and determine the stacking events of the electronic device materials based on the multiple contact positions, the relative positions of the multiple electronic device materials, and their own shapes; determine the theoretical stacking shape of the electronic device materials based on the matching of the stacking events of the electronic device materials and the usage schedule of each electronic device material.

[0021] Step S13: Determine the actual stacking pattern of electronic device materials based on stacking recognition of real-time stored images, and determine the abnormal stacking pattern characteristics of electronic device materials based on the actual stacking pattern and theoretical stacking pattern of electronic device materials.

[0022] Step S14: Mark multiple abnormal stacking locations with abnormal stacking morphology characteristics, and determine the area to be optimized based on multiple abnormal stacking locations and the usage order of the corresponding electronic equipment materials;

[0023] Step S15: Based on the location of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials, respond to the nearby material robot and trigger the material robot to adjust the position of the electronic equipment materials in the area to be optimized.

[0024] In step S11, real-time stored images of electronic device materials are acquired, and the relative positions and shapes of multiple electronic device materials are determined based on material recognition of the real-time stored images.

[0025] In the specific implementation of this invention, the specific steps are as follows:

[0026] S111: Collect the storage location of materials in electronic devices, and determine the real-time storage image of materials in electronic devices based on the dynamic shooting of the storage location of materials in electronic devices by the camera, and determine the background area and material area according to the division of the real-time storage image of materials in electronic devices.

[0027] S112: In the material area, multiple material markers are determined based on the traversal of the material area, and the shape diagram of each electronic device material is determined based on the shape detection of each material marker. The shape of the multiple electronic device materials is determined based on the synchronous recognition of the shape diagram of each electronic device material.

[0028] S113: Determine the current position of each electronic device material based on the position detection of each material marker, and determine the relative position of multiple electronic device materials based on the position comparison of the current positions of each electronic device material, and construct corresponding multimodal data based on the relative positions and shapes of multiple electronic device materials.

[0029] In the embodiments of this application, the storage location of electronic equipment materials is collected. The storage location can be a specific physical coordinate, such as a cell number on a warehouse shelf (e.g., "A-3-2" represents the second cell on the third layer of column A), or it can be a region, such as a rectangular area on the ground. The method of collecting this information is as follows: manual input: the operator selects or inputs the location on the interface; automatic allocation by the system: the WMS automatically allocates the storage location according to the warehousing instruction and transmits the information to the vision system; preset configuration: during system initialization, the field of view of the camera is mapped to the specific storage location / region.

[0030] Once the storage location to be observed is determined, the system will instruct the corresponding camera (which can be fixed or movable) to take a picture of that location. "Dynamic shooting" means that this is not a one-time photo, but a continuous capture of images at a certain frequency (such as a few frames per second or one frame per minute) in order to reflect changes in the state of the material stacking. "Real-time storage image" refers to the image data that the camera has just captured, reflecting the state of the storage location at the current moment. This is usually a digital image file, such as JPG or PNG format, or raw pixel data that is easier to process.

[0031] From the complex images captured by the camera, we separate the "material" part that we are really interested in from the "background" part that we are not interested in, and divide the pixels in the image into different categories, mainly "material" and "background". We enhance the real-time stored image, such as adjusting brightness and contrast, removing noise (e.g., using Gaussian filtering), or performing color space conversion (e.g., from RGB to HSV) to facilitate subsequent segmentation.

[0032] If the material and background differ significantly in color or brightness, a thresholding method can be used. For example, if the material is a specific color (such as blue), a color range can be set, and all pixels within this range in the image can be labeled as "material," with the rest as "background." Edges with drastic brightness changes in the image are detected, and then contour analysis is used to delineate the boundaries of the material. Starting from seed points in the image that belong to the material, the image is expanded outwards according to similarity criteria (such as color and texture) until it cannot be expanded further, thus forming the material region. The output is typically a "mask" image or a segmentation map. In this map, pixels belonging to the "material region" are assigned a specific label (e.g., pixel value 255 or white); pixels belonging to the "background region" are assigned another label (e.g., pixel value 0 or black).

[0033] Furthermore, the predefined "material regions" are typically binary images (masks) where "1" represents a material pixel and "0" represents a background pixel, or a label image containing different material categories obtained through semantic segmentation. "Traversal" refers to finding a continuous set of pixels belonging to the same independent object within the material region. A common technique is "Connected Component Labeling." The algorithm starts with a pixel in the image and checks its neighboring pixels. If a neighboring pixel is also a material (value 1), they belong to the same region and are assigned the same label number (e.g., label 1). This process continues until no more pixels belonging to the region can be found. Then, the next unlabeled material pixel is found, assigned a new label number (label 2), and the process is repeated until all material pixels in the image are labeled. Each independent region found by the connected component labeling algorithm is assigned a unique numerical label (e.g., 1, 2, 3…), which represents an independent material in the image. The number of labels equals the number of independent materials in the image.

[0034] For each individual material tag (i.e., an individual material), the system needs to analyze its pixel distribution and extract features that can describe its shape. These can include: geometric features: area (number of pixels), perimeter (number of outline pixels), bounding box (minimum bounding rectangle), minimum bounding circle, convex hull, centroid (geometric center), etc.; shape descriptors: moments (such as Hu moments, used for shape matching), shape factors (such as roundness, rectangularity), skeleton (extracting the central axis of the material), etc. The "morphological diagram" can be understood as a data structure or feature vector composed of the above-detected morphological features. It is not necessarily a drawn diagram, but rather a description of the shape and size of the material using numbers and parameters. For example, it can be a list or dictionary containing information such as area, perimeter, bounding box size, roundness, etc.

[0035] The process of classifying or identifying using these morphological diagrams involves matching the detected shape features with known material types. The goal is to assign a specific identity or category to each individual material, such as "standard blue phone case," "standard red phone case," "large packaging box," "small packaging box," etc. This typically relies on a predefined rule base or classification model.

[0036] The system can set rules, such as: "If the area is between 100-150 pixels, the color is blue, and the shape factor is close to a rectangle, then it is judged as a standard blue phone case"; the rules can combine multiple features such as color information (if color information is provided in S111 or subsequent steps), size, and shape; if a machine learning model (such as support vector machine or neural network) is used, the model has already learned the feature patterns of different material categories during the training phase; in this step, the model will output a category label based on the input morphological diagram (feature vector); the final determined "self-form" is the specific identity or type of this material individual.

[0037] Knowing the "morphological diagram" of the material represented by each marker; the morphological diagram usually contains key information describing the material's position and shape, most commonly the "centroid" coordinates or the bounding box coordinates; extract information that represents the overall position of the material from the morphological diagram; the most common method is to extract the centroid coordinates of each material; the centroid is the average of all pixel coordinates of the object, which can well represent the center position of the object; if using a bounding box, the coordinates of the upper left and lower right corners of the bounding box, or the coordinates of the center point of the bounding box, are usually extracted; record the extracted coordinate information (such as the centroid coordinates), which is the "current position" of the material in the current image coordinate system, and this position is relative to the pixel coordinate system of the image.

[0038] Analyze the spatial layout relationships between materials; by comparing the coordinates of different materials, their relative positions can be determined; common comparison methods include: distance calculation: calculate the Euclidean distance between the centroids of any two materials; closer distances indicate that they are in contact or close to each other; position determination: by comparing coordinate values, determine the position of one material relative to another; for example, if the x-coordinate of material A is less than the x-coordinate of material B, and its y-coordinate is also less than the y-coordinate of material B, then it can be said that material A is located to the lower left of material B; based on the above comparison results, define the relative positional relationship between each pair of materials or each group of materials, which is a simple positional description (such as "upper left" or "adjacent").

[0039] This introduces the "individual form" of each material and their "relative positions"; combining information about individual materials (individual form) and information about the relationships between materials (relative positions) forms a richer and more comprehensive description. This combined data is called "multimodal data." "Modality" here can be understood as different types of information or data sources; for example, individual form (which is category information, such as "phone case" or "packaging box") and relative position (spatial geometric information) are two different modalities. This information is organized into a structured data format, such as a list or a list of dictionaries; each item in the list corresponds to a material or a group of related materials, including its own form and relative position information with other materials.

[0040] Therefore, the current position of each electronic device material is determined based on the position detection of each material marker, and the relative position of multiple electronic device materials is determined by comparing the current positions of each electronic device material. Based on the relative positions of multiple electronic device materials and their own shapes, corresponding multimodal data is constructed, which takes into account the overall consideration of comparing the current positions of each electronic device material, and ensures the accuracy of the relative positions of multiple electronic device materials.

[0041] At this time, S112 outputs various material markers and their shape diagrams; the shape diagrams contain key information describing the material's position, such as centroid coordinates (x, y); extract the centroid coordinates from the shape diagrams; the centroid is the average of all pixel coordinates of the object, representing the center position of the object; determine the current position: record the extracted centroid coordinates as the current position of the material in the current image coordinate system.

[0042] Calculate the relative positional relationship between any two materials, which can be achieved by calculating the relative distance and direction between their centroid coordinates; for example, Euclidean distance can be used to measure the distance and angles can be used to describe the direction; based on the calculation results, describe the relative positional relationship between the materials, such as "material A is to the lower left of material B" or "the distance between material B and material A is 158.11 pixels", etc.

[0043] The system incorporates the morphological and relative positional information of each material, integrating their morphological characteristics (e.g., type, size) and relative positional information (e.g., distance, direction) into a structured data structure, such as a dictionary or object. It outputs multimodal data, containing the attributes of each material and their spatial relationships. The system not only knows where each material is located but also how they are arranged relative to each other, integrating these different types of information (individual attributes and group relationships) to form higher-level data that can be used for further analysis and decision-making. This multimodal data can be used in subsequent steps, such as S12 analysis of contact points and stacking events, thereby achieving more intelligent material management.

[0044] In step S12, the contact positions between multiple electronic device materials are marked, and the stacking events of the electronic device materials are determined based on the multiple contact positions, the relative positions of the multiple electronic device materials, and their own shapes; the theoretical stacking shape of the electronic device materials is determined based on the matching of the stacking events of the electronic device materials and the usage schedule of each electronic device material.

[0045] In the specific implementation of this invention, the specific steps are as follows:

[0046] S121: Collect the relative positions of multiple electronic device materials, and determine multiple contact nodes based on the position traversal of the relative positions of the multiple electronic device materials. Gradually determine the contact positions between the multiple electronic device materials by autonomously detecting the multiple contact nodes.

[0047] S122: Determine a first sub-stacking event based on the contact position between multiple electronic device materials and the shape of the multiple electronic device materials themselves; determine a second sub-stacking event based on the contact position between multiple electronic device materials and the relative position of the multiple electronic device materials; construct a stacking event of electronic device materials based on the first sub-stacking event and the second sub-stacking event; the stacking event of electronic device materials presents the specific stacking situation of multiple electronic device materials.

[0048] S123: Collect the model numbers of each electronic device material, determine the usage plan table for each electronic device material based on the matching of the model numbers of each electronic device material with the electronic device material database, determine the theoretical stacking position of multiple electronic device materials based on the matching of the stacking events of electronic device materials with the usage plan tables of each electronic device material, and determine the theoretical stacking form of electronic device materials based on the shape of the theoretical stacking position of multiple electronic device materials.

[0049] In embodiments of this application, the relative positions of multiple electronic device materials are collected, and the spatial relationships between the materials are extracted from this information, particularly the degree and direction of their proximity to each other. Optionally, the system will traverse all material pairs (e.g., material A and material B, material A and material C, material B and material C, etc.). For each pair of materials, the system will calculate the geometric relationships between them, which typically includes:

[0050] Distance Calculation: Calculate the Euclidean distance between the center points (or other representative points, such as the geometric center, the lowest point, etc.) of two materials; for example, if the center of material A is at (150, 200) and the center of material B is at (300, 250), then the distance between them is sqrt((300-150)²+(250-200)²)=sqrt(22500+2500)=sqrt(25000)≈158.11 pixels.

[0051] Direction determination: Determine the approximate direction of one material relative to another material (e.g., upper left, upper right, lower left, lower right, or a more precise angle); for example, material A is approximately lower left relative to material B.

[0052] Geometric considerations: Consider the actual shape and size of the material; proximity of the center points does not necessarily mean contact. It is necessary to check whether their edges are in contact, which involves calculating the material's bounding box or a more precise outline.

[0053] The system collects relative positional information (distance, direction, geometry) between material pairs, identifies those that are close enough to have physical contact, and marks these contact points (or areas) as "contact nodes." At this point, the system sets a "contact threshold," which is not a fixed distance but a comprehensive consideration. Larger materials require a greater distance to be considered in contact; pixel distances need to be converted to actual physical distances for evaluation; typical pixel distances for different types of material contact are determined through experiments or machine learning; the system iterates through all material pairs, checking whether their distances and geometric relationships meet the contact conditions; if so, the system attempts to locate the contact location, which is usually not a single point but an area; the system uses image processing techniques (such as Canny edge detection) to extract the edges of material A and material B; on the edge images of the two materials, it searches for areas where pixel values ​​all represent edges (or are very close), these areas are potential contact nodes; these contact areas are marked; for example, they can be marked as "the lower right corner of material A is in contact with the upper left corner of material B."

[0054] If the contact area is blurry, interpolation or other techniques can be used to more accurately locate the contact edges; determine whether it is point contact, line contact, or surface contact; for example, two planar objects form surface contact, while a sphere placed on a plane is point contact; although this is handled in more detail in subsequent steps, it can initially assess whether the contact appears stable (e.g., a larger contact area is more stable); the system refines each contact node to obtain a more precise description of the contact position; for example, the previous area (160-165, 210-215) is refined to "the lower right edge segment L1 of material A and the upper left edge segment L2 of material B contact at point P (approximately 163, 213), forming a line contact."

[0055] Furthermore, the system incorporates the contact positions and shapes of multiple electronic device components to identify basic stacking patterns or postures based on material physical properties and contact methods. Simultaneously, the system analyzes the matching relationship between contact positions and material shapes; it compares the material shapes in the contact areas; for example, the curved edge of material A contacts the straight edge of material B; the system determines whether this contact is "reasonable" or "typical"; a curved object pressing on a flat object is a common stacking method. Based on this shape-contact matching, the system defines a "first sub-stacking event," which describes the basic physical posture of the material stack. If the contact area shapes match well (e.g., plane to plane), it is defined as a "stable support" event; if the contact area shapes do not match (e.g., sharp corner to plane), it is defined as an "unstable contact" event; if the contact occurs in a specific functional area (e.g., the handle of a packaging box is pressed down), it is defined as a "functional area pressure" event.

[0056] By introducing the contact positions and relative positions of multiple electronic device materials, the system identifies the stacking structure or relationship based on the overall spatial layout of the materials and the contact points. Simultaneously, the system analyzes the significance of the contact points within the overall material layout. It examines the position of the contact points relative to the material center, as well as the overall orientation and distance between materials. For example, it considers whether contact point P is close to the bottom of material A and the top of material B, and whether material A is mostly located below material B. Based on this spatial relationship, the system defines a "second sub-stacking event," which describes the structural relationship of the material stacking. Furthermore, if the bottom of one material contacts the top of another material, and the material is mostly below it, it is defined as a "stacked on top" event; if two materials are in side-by-side contact, it is defined as a "side-by-side stacking" event; and if one material leans against another at an angle, it is defined as a "leaning against" event.

[0057] The system introduces a first sub-stacking event and a second sub-stacking event, merging them to form a more comprehensive and specific overall description of the stacking event. At this point, the system combines these two sub-events to form a composite event label or description. This description should simultaneously reflect the physical contact method of the materials and their structural relationship in space. Optionally, the first sub-stacking event is "edge contact stacking," and the second sub-stacking event is "oblique overlap." Combining these two sub-events provides a more complete description of the stacking situation of materials A and B. They are not simply placed flat together, but rather through edge contact, and in an oblique overlap manner. The system constructs the "stacking event of electronic equipment materials" as "oblique edge overlap stacking." This event clearly presents how materials A and B are specifically stacked together: they are through edge contact, and the stacking relationship is one material obliquely overlapping the other.

[0058] Therefore, the model numbers of each electronic device material are collected, and the usage plan table for each electronic device material is determined based on the matching of the model numbers of each electronic device material with the electronic device material database. Based on the matching of the stacking events of electronic device materials with the usage plan tables of each electronic device material, the theoretical stacking position of multiple electronic device materials is determined. Based on the shape of the theoretical stacking position of multiple electronic device materials, the theoretical stacking shape of electronic device materials is determined. This takes into account the overall shape of the theoretical stacking position of multiple electronic device materials, ensuring the accuracy of the theoretical stacking shape of electronic device materials.

[0059] At this point, the model numbers of various electronic equipment materials are collected. The electronic equipment material database is a pre-established database that stores detailed information on various material models, including size, weight, attributes, and most importantly—their usage schedules (e.g., when they are needed, priority, target storage location, processing flow, etc.). The system then matches the collected model numbers with the database. The database returns detailed information on the material model, and the system determines a usage schedule for each material. For example: Material A (“A+”): The usage schedule includes “Priority: Medium”, “Target Storage Area: Area A”, and “Estimated Usage Time: Next Wednesday”; Material B (“B+”): The usage schedule includes “Priority: High”, “Target Storage Area: Area B”, and “Estimated Usage Time: Tomorrow Morning”; Material C (“A+”): The usage schedule is the same as Material A.

[0060] The system collects stacking events, which describe the current actual stacking method of materials; for example, materials A and B are stacked with "diagonal edge overlap". The usage plan contains information such as material priority and target storage area. Based on the material management requirements and the current stacking status, the system infers where these materials should be located if managed according to the plan. At this time, the system compares the current actual stacking status of the materials (stacking events) with the target requirements in its usage plan (such as target storage area and priority). If there is a discrepancy, the system will use the plan to deduce the ideal location. The system determines a "theoretical stacking location" for each material, that is, the location where they should be placed if managed completely according to the usage plan.

[0061] The system combines the theoretical stacking positions and shapes of multiple electronic device materials, visually or structurally describing each material's theoretical position with its shape. For example, material B is in the theoretical position (B area, X2, Y2) and its shape is a "cube"; the system depicts B as an independent cube placed in the upper left corner of B area; material A is in the theoretical position (A area, X1, Y1) and its shape is a "strip"; the system depicts A as horizontally placed in A area; material C is in the theoretical position (A area, X3, Y3) and its shape is also a "strip"; the system depicts C as vertically placed next to A in A area, or horizontally placed below A (if A is vertical); the system integrates this information to construct a "theoretical stacking shape," which is how all materials are combined according to their respective theoretical positions and shapes. This is a hierarchical structure diagram or a three-dimensional model; the system determines the "theoretical stacking shape of electronic device materials"; for example, it is described as: "In A area, a blue phone case is placed horizontally, and another blue phone case is placed vertically below or next to it; in B area, a small packaging box is placed independently."

[0062] Specifically, the system identifies materials A and C as model "A+", and material B as model "B+". The system retrieves information from the database: "A+" has a medium priority and its target area is area A; "B+" has a high priority and its target area is area B. The system analyzes that B has a high priority and its target is in area B, but it is currently being suppressed by A (actually, it is partially obscured by A in area B). Therefore, the theoretical stacking position of B is determined to be a high-priority, unobstructed empty space within area B, such as the upper left corner of area B. The theoretical stacking positions of A and C are determined to be suitable empty spaces within area A. Based on the theoretical positions of A, B, and C and their respective shapes (strips, cubes), the system draws an ideal stacking diagram: B is independently located in the upper left corner of area B, while A and C are arranged horizontally or vertically within area A, without obstructing each other.

[0063] In step S13, the actual stacking pattern of the electronic device material is determined based on the stacking recognition of the real-time stored image, and the abnormal stacking pattern characteristics of the electronic device material are determined according to the actual stacking pattern and the theoretical stacking pattern.

[0064] In the specific implementation of this invention, the specific steps are as follows:

[0065] S131: Acquire real-time stored images and perform stacking recognition on the real-time stored images. During the stacking recognition process, the real-time stored images gradually present each actual stacking contour. The actual stacking shape of the electronic equipment materials is determined based on the synthesis of each actual stacking contour and the position marks of each electronic equipment material.

[0066] S132: Collect the theoretical packing morphology of electronic equipment materials, and mark the corresponding differences based on the comparison between the theoretical packing morphology and the actual packing morphology of electronic equipment materials.

[0067] S133: Based on the traversal of the difference portion, the abnormal positions of a portion of electronic equipment materials are determined, and the abnormal accumulation morphology characteristics of electronic equipment materials are determined based on the synthesis of the abnormal positions of a portion of electronic equipment materials.

[0068] In embodiments of this application, real-time storage images are acquired, which are typically captured in real time by a camera (e.g., an industrial camera or a webcam) mounted above the material storage area. The position and angle of the camera need to be carefully designed to ensure that the area where the materials are stacked can be clearly seen and that the field of view covers the main storage area. The real-time storage images include all visible electronic equipment materials (such as mobile phones, chargers, data cables, packaging boxes, etc.) in the storage area, their current stacking status, and any empty spaces or obstacles present.

[0069] The system needs to identify each individual electronic device material in the image (this relies on the material labels and self-shape recognition capabilities established in S111 and S112). This is usually achieved through object detection algorithms (such as YOLO, SSD, Faster R-CNN based on deep learning). The algorithm will outline each material in the image and give its category (such as "phone case" or "charging cable"). For each detected material, the system will extract its edge contour, which can be achieved through edge detection algorithms (such as the Canny operator) or contour search based on the segmentation results.

[0070] The system needs to determine which materials are stacked together, which can be achieved by analyzing the spatial relationships between the materials: using the method for determining contact nodes in S121, identify which materials have outlines that are in contact with each other or very close; analyze the occlusion phenomenon in the image to determine which materials are on top of other materials; for example, if the image portion of one material is completely occluded by another material, then it can be inferred that the latter is on top of the former; cluster materials that are spatially close to each other, in contact with each other, or have occlusion relationships together to form a "stack group".

[0071] In this process, "real-time stored images gradually present each actual stacking contour during the stacking recognition process," which means that the system will gradually form an understanding of how the materials are stacked together and visually (at least in the internal data structure) outline the overall or local contours formed by these stacks; for example, several stacked phone cases will form an irregular composite contour that is larger than a single phone case.

[0072] Each actual stacking contour and each electronic device material location marker were introduced. Each actual stacking contour represents the visual shape formed by the stacking of materials; the location markers of each electronic device material accurately record the coordinates and orientation information of each material marker (representing a specific material) in the image.

[0073] By combining the "shape of the overall stack" with the "specific position and orientation of individual materials," a more comprehensive and structured description is formed. The system associates each actual stack outline with the individual material markers it contains. For example, the composite outline representing "four stacked phone cases" is associated with the position markers of each of the four phone cases. The system not only knows "there is a stack," but also knows which materials make up the stack and the relative positions and orientations of these materials within the stack. For example, it can be described as: "Stack A consists of material marker 1 (blue phone case, located at the bottom center), material marker 2 (transparent phone case, located slightly to the left above marker 1), material marker 3 (red phone case, located above marker 2), and material marker 4 (white phone case, located slightly to the right above marker 3), presenting an overall tower-like structure with a height of approximately 15cm, located on the second layer of the storage rack, slightly to the left."

[0074] "The actual stacking form of electronic device materials" is a comprehensive description. It not only depicts the overall appearance of the material stack, but also includes the specific identity, location and orientation information of each material that makes up the stack. This is richer and closer to the human intuitive understanding of "a pile of things" than a simple outline or a simple list of individual locations.

[0075] Furthermore, the theoretical stacking form of electronic device materials is collected. The theoretical stacking form of electronic device materials can be represented in various forms, such as a simple textual description (e.g., "maximum of 5 stacked per layer, must be placed horizontally"), a graphical diagram (showing the position and orientation of individual materials), a three-dimensional model containing coordinates and orientation, or a model defined by rules (e.g., constraints based on material size and shape).

[0076] Theoretical stacking data is typically stored in databases, configuration files, or dedicated rule bases, and is associated with specific material types, storage areas, and other information. The system needs to retrieve the corresponding theoretical stacking form based on the material type and storage location currently being processed, and provide a "standard answer" or "expected state" for the actual stacking form, so as to make comparisons and evaluations.

[0077] The system needs to match and compare the theoretical form with the actual form, which usually involves comparing spatial positions, material postures (directions), and stacking layers. For example, the system will compare whether the position of each material in the actual stack is within the theoretically allowed grid, whether the posture is horizontal, and whether the stacking height exceeds the theoretical limit.

[0078] During the comparison process, any deviation from the theoretical form is identified as a discrepancy, including: incorrect location: material is placed in an area that is not allowed by theory; incorrect orientation: material is not placed in the direction required by theory (e.g., tilted, inverted); stacking error: too many or too few layers are stacked, or the stacking method between materials does not conform to the rules; missing / redundant: the actual quantity of material does not match the theoretical expectation; mixing error: other materials that should not be stored in this location are stored.

[0079] Once a difference is identified, the system needs to "mark" it in some way. This could be done by highlighting the difference area or material on an image with a bounding box, color, or text annotation, or by recording the type, location, and material tags involved in the difference in a data structure. The purpose of marking is to clearly indicate where and how the standard has been deviated from.

[0080] Therefore, based on the traversal of the difference portion, the abnormal positions of a portion of electronic equipment materials are determined. The abnormal stacking morphology characteristics of electronic equipment materials are determined by synthesizing the abnormal positions of a portion of electronic equipment materials. This method takes into account the overall consideration of synthesizing the abnormal positions of a portion of electronic equipment materials, ensuring the accuracy of the abnormal stacking morphology characteristics of electronic equipment materials. At the same time, a theoretical stacking morphology is introduced, realizing the comparison between the actual stacking morphology and the theoretical stacking morphology of electronic equipment materials. This enables the feature extraction of the abnormal stacking morphology characteristics of electronic equipment materials, improving the accuracy of the abnormal stacking morphology characteristics of electronic equipment materials.

[0081] At this point, the variance component is introduced, which includes material orientation errors (such as tilting or inverting), position errors (such as exceeding the specified area), stacking quantity errors, stacking method errors, etc., and is usually already associated with specific material tags. The system needs to traverse (check one by one) these tagged variances. For each variance, the system needs to determine which material tags it specifically affects. For example, a variance tag of "orientation error: tilting" corresponds to material tag 2; a variance tag of "stacking error: too many layers" corresponds to material tags 1, 2, 3, and 4 (because they together constitute this erroneous stack).

[0082] For each affected material tag, the system needs to further determine its "abnormal location." Here, "location" refers not only to coordinates but also, more importantly, to its position or state relative to its "normal" state. The abnormal location can be specifically described as:

[0083] Abnormal posture location: For example, "Material 2 is located in the upper right corner of its marked area and tilted at an angle of 15 degrees"; Abnormal position deviation: For example, "Material 3 is located outside its marked area, with specific coordinates (X,Y)"; Abnormal stacking position: For example, "Material 4 is located directly above Material 1, forming a non-standard vertical stack"; Crowded / squeezed position: For example, "Material 5 is squeezed into a corner by Materials 6 and 7 and cannot be removed normally".

[0084] The general difference markers in S132 are transformed into precise descriptions of specific materials and their specific abnormal states. These descriptions will guide subsequent robot operations or human intervention. The system needs to "synthesize" these scattered abnormal location information for overall analysis. This is not merely a simple listing, but rather understanding how these anomalies interact to form a specific, describable "morphology." Based on the results of the synthetic analysis, the system extracts one or more features that can summarize the current abnormal accumulation state. These features should be more macroscopic than the abnormal location of a single material and better reflect the overall problem. Abnormal accumulation morphology features include:

[0085] Local tilting / unstable structure: For example, "A tilted, unstable material pile was formed in areas A1-A3"; Over-stacking area: For example, "More than three layers of high stacking occurred in area B2"; Material squeezing zone: For example, "A congested area where materials are squeezed was formed near the conveyor belt inlet"; Abnormal posture aggregation area: For example, "Multiple materials with different degrees of tilt were observed in corner area C3"; Specific error pattern: For example, "A common error stacking pattern was detected: small materials were placed on top of large materials, resulting in instability".

[0086] Extracting specific, scattered anomalies into a describable, representative anomaly accumulation pattern or state helps to understand the problem more efficiently and provides guidance for subsequent decisions (such as which area to prioritize and how to adjust it).

[0087] In step S14, multiple abnormal stacking locations with abnormal stacking morphology characteristics are marked, and the area to be optimized is determined based on the multiple abnormal stacking locations and the usage order of the corresponding electronic equipment materials.

[0088] In the specific implementation of this invention, the specific steps are as follows:

[0089] S141: Real-time monitoring of abnormal stacking morphology characteristics, determination of corresponding abnormal stacking distribution map based on the detection of abnormal stacking morphology characteristics, marking of multiple abnormal stacking locations in the abnormal stacking distribution map, and presentation of electronic equipment materials corresponding to multiple abnormal stacking locations.

[0090] S142: Based on the traceability of electronic equipment materials corresponding to multiple abnormal stacking locations, a corresponding usage plan is determined, and the usage order of multiple electronic equipment materials is determined according to the matching of the usage plan. At the same time, multiple attitude parameters are determined based on the attitude detection of electronic equipment materials corresponding to multiple abnormal stacking locations, and the current attitude of the electronic equipment material is determined according to the multiple attitude parameters and the shape of the corresponding electronic equipment material.

[0091] S143: Determine the first sub-optimization region based on multiple abnormal stacking locations and the current posture of each electronic device material; determine the second sub-optimization region based on the usage order of the electronic device materials corresponding to the multiple abnormal stacking locations; and determine the region to be optimized based on the combination of the first and second sub-optimization regions.

[0092] In the embodiments of this application, the system will not detect the anomaly only once, but will continuously monitor the abnormal stacking morphology characteristics determined in S133. This can be achieved through continuous video monitoring, sensor data reading or periodic image acquisition. The system needs to be able to capture whether the abnormal morphology has changed (e.g., whether the tilted material has become more tilted, or whether new material has been added to the abnormal stack).

[0093] After creating the abnormal stacking distribution map, the system needs to clearly mark which specific locations have abnormalities on the map. This is usually achieved by placing markers (such as dots, boxes, and color blocks) on the map. At the same time, the system needs to associate these markers with the specific materials previously identified, so that people viewing the distribution map can intuitively know which material the marker represents and its abnormal situation. The system needs to display the abnormal stacking distribution map with markers and material information to relevant personnel (such as monitoring personnel and robot control programs) or record it. The presentation method can be screen display, printed report, or data interface output.

[0094] Optionally, the system continuously monitors the excessive stacking height in areas A1-A3 and the tilting of M2 and M4, as well as the out-of-bounds position of M5; if the camera captures the tilt angle of M2 from…, it presents the electronic equipment materials corresponding to multiple abnormal stacking positions; on the two-dimensional grid distribution map we created earlier:

[0095] In the cells representing areas A1-A3, fill in the cells with red or mark them with a red cross, and add the text "Height Exceeds Limit Stacking (Involving M1-M4)"; mark the point or small area representing the location of M2 with a yellow circle, and add the text "Material M2: Tilted"; also mark the point or small area representing the location of M4 with a yellow circle, and add the text "Material M4: Tilted"; mark the point or small area representing the location of M5 with a blue arrow pointing in the direction of its deviation, and add the text "Material M5: Location Exceeds Limit".

[0096] The system's monitoring screen will display this two-dimensional grid distribution map in real time; the map clearly marks the red areas (A1-A3, M1-M4), yellow dots (M2, M4), and blue arrows (M5); there will be a legend explaining the meaning of each mark; operators or automated systems can see at a glance which materials are abnormal in which locations.

[0097] Furthermore, based on the traceability of electronic equipment materials corresponding to multiple abnormal stacking locations, a corresponding usage plan is determined, and the usage order of multiple electronic equipment materials is determined according to the matching of the usage plan. At the same time, based on the attitude detection of electronic equipment materials corresponding to multiple abnormal stacking locations, multiple attitude parameters are determined, and the current attitude of the electronic equipment material is determined according to the multiple attitude parameters and the shape of the corresponding electronic equipment material. This overall consideration of multiple attitude parameters and the shape of the corresponding electronic equipment material ensures the accuracy of the current attitude of the electronic equipment material.

[0098] At this point, each electronic device material typically has a unique identifier (such as a barcode, RFID tag, serial number, etc.). When the system identifies an abnormal accumulation at a certain location (such as area A1 marked in S141) and knows that this area involves materials M1, M2, and M3, the system will use the unique identifiers of these materials to query the database or material management system. This system will record information related to each material, including a "usage plan". The usage plan is a simple list, but also a complex scheduling plan that specifies when, where, and for what production task or order the material will be used. Traceability is the process of finding this planning information through the material ID.

[0099] After obtaining the usage plans for each material, the system needs to determine a priority based on these plans. The most common way is to sort them by "planned usage time". The earlier the planned usage time of a material, the higher its priority. This sorting result is the "usage order". Of course, there are other matching rules, such as based on the importance and urgency of the material, but time order is the most intuitive and commonly used.

[0100] Optionally, suppose that in S141, the system marks an anomaly in area A1, involving materials W11, W12, and W13; the system connects to the warehouse management system (WMS) by scanning or recognizing the barcodes of W11, W12, and W13; the WMS records: W11: Order #1234 scheduled for production line A at 2 PM today; W12: Order #2345 scheduled for production line B at 9 AM tomorrow; W13: Order #3456 scheduled for production line C the day after tomorrow.

[0101] According to the planned times recorded in WMS: W11: 2 PM today (earliest); W12: 9 AM tomorrow; W13: the day after tomorrow; therefore, the system determines the order of use as: W11>W12>W13. This means that when dealing with abnormal accumulation in the A1 area, W11 should be prioritized for easy access, followed by W12, and finally W13.

[0102] Sensors (typically vision sensors, such as cameras) are used to precisely measure the specific state of materials in abnormal stacking; the goal of attitude detection is to obtain quantitative data describing how the materials are placed, namely "attitude parameters". These parameters include: rotation angle: how many degrees the material has rotated around its central axis (e.g., whether a mobile phone is lying flat or upright); tilt angle: the angle between the material and a horizontal or vertical plane (e.g., how many degrees is a box tilted); position offset: the amount of offset of the material's center point relative to its normal storage position (e.g., whether the material has exceeded the boundary).

[0103] Attitude parameters are the raw data, while "current attitude" is a more easily understood state description after interpreting and describing these parameters; the system needs to combine the physical form of the material (e.g., W12 is a cuboid power bank, W13 is a line) to interpret the meaning of these parameters; for example, a cuboid tilted at 30 degrees means instability, while a tangled line means it is difficult to use.

[0104] Optionally, the system already knows that W11, W12, and W13 are abnormally piled up in area A1. Now, the system uses the camera to take detailed pictures and analyze these materials: For W11 (a phone case): it is detected that it is pressed under W12 and cannot be seen directly; however, based on the obstruction relationship and the posture of W12, it can be inferred that W11 is lying flat but is compressed very flat; For W12 (a power bank): it is detected that it is tilted at about 30 degrees, the bottom is slightly raised, and its long side forms a 45-degree angle with the long side of the storage cell; the posture parameters are: tilt angle = 30°, rotation angle = 45°, and part of the bottom is suspended; For W13 (a data cable): it is detected that it is wrapped around other materials, and part of it is suspended outside the edge of the storage cell; the posture parameters are: wrapped state, suspended length = 5cm.

[0105] W11: The posture parameters are inferred as "flattened, lying flat"; combined with its shape (phone cases are usually thin and flat), the current posture is described as "deformed by pressure, difficult to remove"; W12: The posture parameters are "tilted 30°, rotated 45°, bottom suspended"; combined with its shape (rectangular power bank), the current posture is described as "severely tilted, unstable, partially suspended"; W13: The posture parameters are "entangled, 5cm protruding"; combined with its shape (data cable), the current posture is described as "entangled on other materials, partially protruding, difficult to quickly remove".

[0106] Therefore, a first sub-optimization region is determined based on multiple abnormal stacking locations and the current posture of each electronic device material. A second sub-optimization region is determined based on the usage order of the electronic device materials corresponding to the multiple abnormal stacking locations. The region to be optimized is determined based on the synthesis of the first and second sub-optimization regions. This approach takes into account the overall consideration of the synthesis of the first and second sub-optimization regions, ensuring the accuracy of the region to be optimized.

[0107] At this point, the input is a list of all abnormal accumulation locations identified in the previous step (such as S141), and the specific "current posture" of each abnormal material determined in S142 (e.g., flattened, tilted, entangled, inverted, etc.); the system will evaluate the posture of each abnormal accumulation location and its material according to preset rules or priorities; for example:

[0108] Highest priority: Materials at risk of falling (e.g., severely tilted, with edges hanging in the air), damaged (e.g., crushed by heavy objects, deformed by squeezing), or blocking passageways (e.g., material piles blocking important passageways);

[0109] Second highest priority: Although the material's posture does not directly pose a danger, it is very unfavorable for quick retrieval (such as being completely covered or entangled with other materials).

[0110] Lower priority: abnormal posture but relatively safe and does not affect retrieval (e.g., slight misplacement).

[0111] Based on the above assessment, the system will filter out the "worst" or "most dangerous" locations from all abnormal accumulation locations and aggregate them into one or more "first sub-optimization regions". These regions are the ones that need to be physically intervened or adjusted first.

[0112] Furthermore, the input is a list of all abnormal stacking locations, and the usage order of each abnormal material determined in S142 (i.e., which material needs to be used first and which needs to be used later); the system sorts the abnormal stacking locations according to the usage order of the materials; those abnormal stacking locations containing the materials that are "most needed first" are given higher priority; based on the priority of the usage order, the system filters out those locations that "contain the most urgently needed materials" from all abnormal stacking locations and aggregates them into one or more "second sub-optimization areas", which are areas that need to be prioritized to meet the timely supply demand.

[0113] A first and second sub-optimization region were introduced to integrate two sets of priorities: one based on "physical state urgency" and the other on "usage time urgency." This resulted in a comprehensive and clear list of "regions to be optimized." This integration process involved:

[0114] Union: Include all positions in the two sub-optimization regions to form a complete list of regions to be optimized. This means that all regions that need to be processed due to state differences, as well as all regions that need to be processed urgently, will be included.

[0115] Intersection: Only select those positions that appear in both sub-optimization regions at the same time. This means that only those regions that are both dangerous / difficult to select and urgently needed will be prioritized. This strategy is the most stringent.

[0116] Weighted merging / priority sorting: Assign different weights to the two sub-optimization areas (e.g., physical security has a higher weight), or merge the two lists and sort them according to a comprehensive score (combining status and usage order) to determine the final priority order. This is the most flexible and commonly used method, outputting a final list of "areas to be optimized" (with priority sorting). This list clearly tells the subsequent material handling or robot operation where to focus first and where to focus later.

[0117] Specifically, assume the warehouse contains the following abnormal accumulation locations and their information (from S141 and S142):

[0118] Area A1:

[0119] Materials: W11 (attitude: flattened, lying flat), W12 (attitude: tilted at 30°, unstable), W13 (attitude: wrapped, hanging 5cm);

[0120] Usage order: W11 (to be used this afternoon) > W12 (to be used tomorrow) > W13 (to be used the day after tomorrow);

[0121] Area B2:

[0122] Materials: W21 (attitude: slightly tilted at 5°), W22 (attitude: inverted);

[0123] Usage order: W21 (to be used next Monday) > W22 (to be used next Wednesday);

[0124] C3 area:

[0125] Materials: W31 (Attitude: Completely covered by other materials), W32 (Attitude: Normal);

[0126] Usage order: W31 (to be used tomorrow) > W32 (to be used next Friday).

[0127] Determine the first sub-optimization region: Evaluate A1: W12 is tilted and unstable (high risk), W13 is entangled and suspended (medium risk), W11 is flattened (damaged, medium risk); A1 is in very poor overall condition; Evaluate B2: W22 is inverted (damaged, medium risk), W21 is slightly tilted (low risk); B2 has one medium risk point; Evaluate C3: W31 is completely covered (difficult to access, low risk); C3 has one difficult-to-access point; Result: Region A1 is determined as the first sub-optimization region because it contains multiple high-risk / medium-risk points; B2 and C3 are not included or have lower priority due to their relatively low risk.

[0128] Determine the second sub-optimization region: Evaluate A1: W11 to be used this afternoon (highest priority), W12 to be used tomorrow (second highest priority), W13 to be used the day after tomorrow (medium priority); A1 contains the highest priority material W11; Evaluate B2: W21 to be used next Monday (medium priority), W22 to be used next Wednesday (lower priority); B2's usage order is not urgent; Evaluate C3: W31 to be used tomorrow (second highest priority), W32 to be used next Friday (lower priority); C3 contains the second highest priority material W31; Result: A1 (because of W11) and C3 (because of W31) are determined as the second sub-optimization region.

[0129] Synthesis strategy (assuming weighted merging / priority sorting): A1: High score in the first sub-region (physical state), and also the highest score in the second sub-region (usage order); highest overall score; C3: Low score in the first sub-region, but the second highest score in the second sub-region (because W31 will be used tomorrow); second highest overall score; B2: Low scores in both sub-regions; Result: The regions to be optimized are determined to be {A1, C3}, and the optimization priority of A1 is higher than that of C3; Although there are anomalies in region B2, it is temporarily placed behind because the physical state risk is low and the use is not urgent.

[0130] Through S143, the system no longer just knows where things are messy, but clearly knows "where to start organizing". In this example, the system will prioritize A1 area because it has both physical hazards (W12, W13) and contains the most urgently needed material (W11). Next, it will handle C3 area because although the physical condition is not a big problem, W31 is needed tomorrow and is difficult to retrieve because it is at the bottom. This provides clear instructions for subsequent automated organization or manual intervention.

[0131] In step S15, based on the location of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials, the nearby material robot is responded to and the material robot is triggered to adjust the position of the electronic equipment materials in the area to be optimized.

[0132] In the specific implementation of this invention, the specific steps are as follows:

[0133] S151: Based on the location detection of the area to be optimized, the location of the area to be optimized is determined, and the location of the area to be optimized is subjected to peripheral detection to mark the electronic equipment materials located in the peripheral location of the area to be optimized. At this time, the electronic equipment materials located in the peripheral location of the area to be optimized are subjected to morphological detection, and the stacking morphology of the peripheral electronic equipment materials is determined.

[0134] S152: Determine a first stacking optimization path based on the regional shape of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials; determine a second stacking optimization path based on the regional location of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials.

[0135] S153: Collect the material robots adjacent to the area to be optimized, mark the current position of the material robots, determine the material optimization path of the material robots for electronic equipment materials in the area to be optimized based on the current position of the material robots, the combination of the first stacking optimization path and the second stacking optimization path, and trigger the material robots to adjust the position of electronic equipment materials in the area to be optimized based on the execution of the material optimization path.

[0136] In the embodiments of this application, the position of the region to be optimized is determined based on the position detection of the region to be optimized, and the position of the region to be optimized is subjected to peripheral detection to mark the electronic equipment materials located in the peripheral position of the region to be optimized. At this time, the electronic equipment materials located in the peripheral position of the region to be optimized are subjected to shape detection to determine the stacking shape of the peripheral electronic equipment materials, which is compatible with the overall consideration of the position detection of the region to be optimized and ensures the accuracy of the position of the region to be optimized.

[0137] At this point, in S143, we have determined the approximate range of the "area to be optimized" (such as a certain compartment on a shelf or a certain area on the ground); now, the system needs to know the specific coordinates of this area in physical space more precisely, which usually requires combining information from multiple sensors.

[0138] The system uses cameras (to identify markings on shelves or area boundaries), LiDAR (to accurately measure the relative distance between the area boundary and the robot itself), inertial measurement units (IMUs, which help the robot understand its own posture changes), and pre-built maps or warehouse layout information. Through SLAM (Simultaneous Localization and Mapping) or localization algorithms based on known maps, the system calculates the precise boundary coordinates of the area to be optimized. For example, it determines that area A1 is a rectangle with the coordinates of its lower left corner (x=1.2m, y=0.8m) and upper right corner (x=1.5m, y=1.1m). The system internally updates the precise geographic information of area A1 to be optimized and displays it visually on the operation interface.

[0139] The system sets a "periphery" detection range, such as the area within 0.3 meters of the boundary of area A1. This range can be adjusted according to the actual situation to ensure that nearby materials that may affect the robot's entry or operation can be detected. The system then uses sensors such as cameras and LiDAR to scan this periphery area again. The image recognition algorithm analyzes the image to detect which are electronic equipment materials and which are shelves, walls, or other fixed obstacles. For the identified electronic equipment materials located within the periphery, the system will label them as "peripheral materials" and record their location and ID. For example, if the system detects a material W14 on the north side of area A1 (y=1.1m to y=1.4m, x=1.2m to x=1.5m) and a material W15 on the east side (x=1.5m to x=1.8m, y=0.8m to y=1.1m), the system will label W14 and W15 as periphery materials of area A1.

[0140] The system focuses on the surrounding materials (W14 and W15) marked in sub-step 2. It uses a camera (combined with depth information, such as binocular vision or structured light) to carefully observe the shape, orientation, and stacking method of these materials. The system analyzes the morphology of these surrounding materials: W14: The system detects that W14 is a cuboid, currently placed vertically, and relatively stably leaning against the north boundary of area A1; W15: The system detects that W15 is a flat, circular plate-like object, leaning diagonally against the east boundary of area A1, which is prone to shaking or falling. The system records the morphology of W14 as "vertically stable leaning against" and the morphology of W15 as "diagonally leaning against and unstable", which forms a more complete description of the surrounding environment.

[0141] Optionally, the system precisely knows that the boundary of area A1 is (1.2m, 0.8m) to (1.5m, 1.1m); the system scans the area 0.3 meters outside the boundary of area A1 and finds material W14 on the north side and material W15 on the east side, and marks them as the surrounding materials of A1; the system further detects that W14 is vertically stable and placed, while W15 is tilted and unstable. This information is the basis for the next step (S152 planning and optimizing the path); for example, the robot knows that it cannot directly collide with W15 from the east side, and needs to go around W14 from the north side (if W14 is stable enough and the passage is sufficient), or choose to enter from the southwest corner; the understanding of the instability of W15 also prompts the robot to be more careful during operation to avoid causing W15 to move; S151 lays the foundation for the safe and efficient execution of subsequent operations.

[0142] Furthermore, a first stacking optimization path is determined based on the regional shape of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials, and a second stacking optimization path is determined based on the regional location of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials. This approach takes into account both the regional location of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials, ensuring the accuracy of the second stacking optimization path.

[0143] At this point, the shape of the region to be optimized is collected. The shape of the region refers to the physical characteristics of the region itself. Optionally, it can be a regular rectangular grid or an irregular corner, its size, and whether there are fixed partitions or obstacles inside. For example, region A1 is a standard 1m x 0.5m rectangular grid.

[0144] The data collection process involves collecting the stacking patterns of surrounding electronic equipment materials. The stacking patterns refer to the state of materials adjacent to the boundary of the area to be optimized. Optionally, these materials may be neatly stacked, randomly placed, tall or short, or may obstruct the passage into the area to be optimized. For example, in the case of S151, W14 on the north side is placed vertically and stably, while W15 on the east side is placed diagonally and unstable.

[0145] The morphology of the area determines the maximum range and posture of the robot's operation inside; the accumulation pattern of the surrounding materials (especially the instability and position of W15) directly affects how the robot can safely approach and enter the area; the path needs to avoid the unstable W15 and consider how to deal with the stable "neighbor" W14; the path also needs to consider how to deal with the most "troublesome" materials inside the area first (such as the abnormal materials identified in S143, such as W11 which is pressed at the bottom). This path is more like a "tactical" level plan, focusing on how to safely and effectively enter and start sorting the internal materials in a given physical space and surrounding environment. It is a route that goes around the boundary of the area, avoids obstacles, and finally reaches the target material inside.

[0146] Optionally, A1 is a 1m x 0.5m rectangular grid; W14 on the north side is vertically stable; W15 on the east side is diagonally unstable. The first optimal stacking path is determined as follows: due to the instability of W15 on the east side, the robot cannot approach from the east; although W14 on the north side is stable, it is close to the boundary, requiring the robot to carefully bypass it or pass through its narrow side space; the safest entry method is from the southwest or northwest corner; assuming the southwest corner is chosen, the path is planned as follows: the robot moves along the ground (assuming it is under or beside the shelf) to the southwest corner of A1; the robot moves slightly north to avoid the bottom of W14 (if W14 is close to the south boundary); the robot enters the A1 area and moves a short distance north along the west boundary; reaching the predetermined position, it prepares to begin processing the materials inside A1 (e.g., first attempting to lift the bottom-most-pressed W11). The first optimal stacking path clearly defines how the robot can safely enter the A1 area from the outside and reach a position where it can handle the internal materials, while avoiding the unstable W15 and the W14 that requires careful handling.

[0147] For the second stacking optimization path, the regional location of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials are introduced. The regional location of the area to be optimized refers to the macroscopic location of the area to be optimized in the entire material area (such as the entire warehouse, workshop or shelf area). Is it close to the entrance / exit? Is it close to other material areas that require frequent operation? Is it easily accessible by robots (or other handling equipment)? For example, area A1 is located in the middle layer of the shelf and is 1.5 meters away from the main aisle. The stacking pattern of the surrounding materials is the state of the surrounding materials detected in S151. They not only affect the entry method, but also the path after the materials are moved out or repositioned. The planning of the second stacking optimization path focuses more on "material movement and relocation from a global perspective".

[0148] The location of a region determines the total cost (time, energy) for a robot or other equipment to reach that region; optimization is not just about sorting, but also about moving certain materials to more suitable locations; the location of a region will affect the selection of these target locations; for example, if W11 needs to be used preferentially, should it be moved to an area that is easier to access? Path planning needs to consider whether it will affect the normal access to other materials, or whether existing material flows (such as the paths that other robots are moving along) can be utilized. This path is more like "strategic" planning, focusing on the rational flow and final placement of materials throughout the material area, and how to complete these operations efficiently, taking into account the overall layout.

[0149] Optionally, A1 is located in the middle layer of the shelf, 1.5 meters from the main aisle; W14 on the north side is stable; W15 on the east side is unstable; determine the second stacking optimization path: A1 is a certain distance from the main aisle, meaning the robot needs to travel a certain distance to reach it; optimization not only needs to solve the problems inside A1, but also needs to consider the storage location of the optimized materials (especially W11); the instability of W15 also means that when moving materials in A1, we need to consider whether it will affect W15; starting from the main aisle, how to reach the southwest corner of A1 as quickly as possible (based on the starting point of the first path), and how to leave A1 to return or go to the next task point after the operation is completed; for example, plan a complete route from the main aisle -> southwest corner of A1 -> (operation) -> southwest corner of A1 -> main aisle; for W11, if it needs to be used first, the second path suggests moving it into A1 after sorting. For W12 and W13, it is recommended to move them to an outer, more accessible location, or even to another area closer to the main aisle if space in A1 allows and the adjacent area is suitable (assuming the system has the necessary permissions and planning capabilities). For W12 and W13, it is recommended to stack them more neatly to leave space. When planning the path, it is necessary to consider minimizing disturbance to the unstable W15 when moving materials in A1. When operating W11, it is necessary to fix or move W15 first, but this is a more complex operation, and here we assume that the risk is considered when planning the movement route. The second stacking optimization path not only includes the macro route of how to enter and leave A1, but also includes strategic suggestions on where W11 should ultimately be placed in A1, and how to minimize the impact on the surrounding environment (such as W15) throughout the process. It is more like a "task plan" that includes a start point, end point, waypoints, and operational suggestions.

[0150] The first stacking optimization path solves the problem of "how to accurately and safely reach and begin sorting the target area"; the second stacking optimization path solves the problem of "how to efficiently and reasonably complete this sorting within the entire material area, while considering the long-term placement of the materials". These two paths are not isolated; the final operation path executed by the robot is a "combination" of the first and second stacking optimization paths. For example, the robot will follow the overall movement framework planned by the second path, but after entering area A1, it will switch to the detailed operation steps planned by the first path. These two paths together ensure that the optimized operation considers both local details and meets the requirements of global efficiency and safety.

[0151] Therefore, the material robots adjacent to the area to be optimized are collected, and their current positions are marked. Based on the current position of the material robot and the synthesis of the first and second stacking optimization paths, the material optimization path of the material robot for the electronic equipment materials in the area to be optimized is determined. Based on the execution of the material optimization path, the material robot is triggered to adjust the position of the electronic equipment materials in the area to be optimized. This approach takes into account the overall consideration of the current position of the material robot, the synthesis of the first and second stacking optimization paths, and ensures the accuracy of the material optimization path of the material robot for the electronic equipment materials in the area to be optimized. At the same time, the introduction of the area to be optimized enables further control over the area to be optimized, thereby responding to the nearby material robots and adjusting the position of the electronic equipment materials in the area to be optimized through the cooperation of the material robots, so as to achieve the rational stacking of electronic equipment materials.

[0152] At this point, the system collects information on the material handling robots near the area to be optimized. The system needs to know which material handling robots (usually AGVs, AMRs, or other automated handling equipment) are idle or available for use near the area to be optimized (e.g., A1). This is usually achieved through the factory's equipment management system or network broadcast. The system will select the most suitable robot to perform the task based on factors such as the type of robot (some are more suitable for handling heavy materials, while others are more flexible), current battery level, load capacity, and distance from the area to be optimized. For example, if material W11 in area A1 is heavy, the system will select an AGV with a strong load capacity; if flexible detour is required, an AMR will be selected.

[0153] Optionally, in our example of area A1, the system scans for three material handling robots near A1: robot R1 (an AGV with a load capacity of 100kg, currently 5 meters southwest of A1), robot R2 (an AMR with a load capacity of 50kg, currently 3 meters north of A1), and robot R3 (an AGV with a load capacity of 200kg, currently performing other tasks in area B2, which is farther away). After evaluation, the system selects R1 because it has sufficient load capacity and a suitable distance, or selects R2 because it is more flexible and can better execute the complex path planned in S152. Let's assume the system selects R1.

[0154] Once a robot (e.g., R1) is selected, the system needs to know R1's current location and orientation precisely. This is typically achieved by combining the robot's own sensors (such as wheel speedometers and IMUs) with external positioning systems (such as LiDAR SLAM, QR code navigation, and visual positioning). The location information is updated in real time to the system's database or control center to ensure that subsequent path planning is based on the latest and most accurate data. Optionally, the system can use R1's built-in LiDAR and the factory's SLAM map to determine R1's current precise location as coordinates (X=10.2, Y=15.5) and orientation due east. This information is recorded for the next step of path synthesis.

[0155] The system collects the precise starting position and orientation of robot R1, the first stacking optimization path planned in S152 (a fine route and internal operation suggestions for entering area A1), and the second stacking optimization path planned in S152 (considering the global macroscopic movement framework and material final position suggestions). The system needs to combine these three to generate a complete, continuous, and feasible robot motion trajectory. This is not just a simple splicing, but also needs to consider: the transition from R1's current position to the starting point of the first path; the consistency of the first and second paths in terms of the operation target (such as moving W11); ensuring that the synthesized path avoids all known obstacles, including the surrounding materials marked in S151 (such as W14, W15); converting the path into specific robot actions, such as moving forward, turning, stopping, extending the robotic arm, grasping, lifting, moving, and putting down; generating the final path: the final output "material optimization path" is a series of precise instructions that guide R1 on how to move from its current position to area A1, grasp W11, move it to a new, optimized position, and then ( ) move R1 to a safe position or standby state.

[0156] Optionally, the system calculates the following optimized material path for R1: 1. Move 3 meters southeast from the current position (X=10.2, Y=15.5) to the western entrance point of area A1 (X=12.5, Y=16.0); 2. Slowly move 0.5 meters south along the western edge of area A1 to the vicinity of W11's current position (X=12.5, Y=16.5); 3. Stop, adjust the orientation, and align the robotic arm with W11; 4. Extend the robotic arm and grab W11; 5. Retract the robotic arm and raise W11 to a safe height of 0.3 meters; 6. Following the internal suggestion of the first path, move 0.4 meters north to the new position of W11 planned in S152 (X=12.5, Y=16.9); 7. Lower and place W11 in the new position; 8. Retract the robotic arm; 9. Move 0.5 meters west to leave area A1.

[0157] This path integrates the starting point of R1, the fine-grained operations of the first path (grabbing and placing the specific location of W11), the macro framework of the second path (entering from the west and eventually placing W11 in a new location on the north side), and ensures that the unstable W15 on the east side is avoided.

[0158] The system sends the "material optimization path" (i.e., a series of precise instructions) generated in the previous step to the selected material robot (R1). After receiving the instructions, R1's control system drives its own motors, joints, and other actuators to complete the movement, grasping, and placement actions step by step according to the instructions. During the execution, R1's sensors continuously provide feedback on its position, posture, and grasping status. The control system makes minor adjustments based on the real-time feedback to cope with unexpected small deviations (for example, W11's actual position may be slightly off-target). Once R1 has completed all the instructions and placed W11 in the new optimized position, the position adjustment is complete. The system can record this operation and update W11's position information.

[0159] Optionally, upon receiving the aforementioned 9-step instructions, R1 begins to move. The LiDAR scans the surrounding environment in real time to ensure it does not collide with W14 and W15. The robotic arm precisely grasps W11 and moves it to the position (X=12.5, Y=16.9) according to the path before placing it down. After completion, R1 returns to standby mode. The system updates the database, marking that W11 has been moved to the new position. The abnormal accumulation problem in area A1 is partially resolved (at least W11 is correctly placed; further operations are needed to handle other materials or completely organize the area).

[0160] Please see Figure 2 , Figure 2 This is a schematic diagram of the structural composition of the material accumulation morphology feature extraction system in an embodiment of the present invention; the material accumulation morphology feature extraction system includes:

[0161] The material identification module 21 is used to acquire real-time stored images of materials in electronic devices, and to determine the relative positions and shapes of multiple electronic device materials based on the material identification of the real-time stored images.

[0162] The theoretical stacking pattern module 22 is used to mark the contact positions between multiple electronic device materials, and to determine the stacking events of electronic device materials based on the multiple contact positions, the relative positions of the multiple electronic device materials, and their own shapes; and to determine the theoretical stacking pattern of electronic device materials based on the matching of the stacking events of electronic device materials and the usage schedule of each electronic device material.

[0163] The abnormal stacking morphology feature module 23 is used to determine the actual stacking morphology of electronic equipment materials based on stacking recognition of real-time stored images, and to determine the abnormal stacking morphology features of electronic equipment materials based on the actual stacking morphology and theoretical stacking morphology of electronic equipment materials.

[0164] The optimization area module 24 is used to mark multiple abnormal stacking locations with abnormal stacking morphology characteristics, and to determine the optimization area based on the multiple abnormal stacking locations and the usage order of the corresponding electronic equipment materials.

[0165] The position adjustment module 25 is used to respond to the nearby material robot based on the position of the area to be optimized and the stacking pattern of the surrounding electronic equipment materials, and to trigger the material robot to adjust the position of the electronic equipment materials in the area to be optimized.

[0166] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for extracting material pile morphology features, characterized in that, The application comprises: Collecting real-time storage images of electronic device materials, determining relative positions and self shapes of the electronic device materials based on material recognition of the real-time storage images; Marking contact positions between the electronic device materials, and determining a stacking event of the electronic device materials according to the contact positions, the relative positions and the self shapes of the electronic device materials; Determining a theoretical stacking shape of the electronic device materials according to matching of the stacking event and a use schedule of each electronic device material, comprising: collecting the relative positions of the electronic device materials, and determining contact nodes according to position traversal of the relative positions, and gradually determining contact positions between the electronic device materials along autonomous detection of the contact nodes; determining a first sub-stacking event based on the contact positions and the self shapes of the electronic device materials, determining a second sub-stacking event based on the contact positions and the relative positions of the electronic device materials, constructing the stacking event of the electronic device materials based on the first sub-stacking event and the second sub-stacking event, and the stacking event of the electronic device materials representing specific stacking conditions of the electronic device materials; collecting models of each electronic device material, determining the use schedule of each electronic device material according to matching of the models and an electronic device material database, and determining a theoretical stacking position of the electronic device materials based on matching of the stacking event and the use schedule, and determining the theoretical stacking shape of the electronic device materials according to the self shapes of the theoretical stacking positions; Determining an actual stacking shape of the electronic device materials based on stacking recognition of the real-time storage images, and determining an abnormal stacking shape feature of the electronic device materials according to the actual stacking shape and the theoretical stacking shape; Marking multiple abnormal stacking positions of the abnormal stacking shape feature, and determining a to-be-optimized area according to the abnormal stacking positions and a use order of corresponding electronic device materials; Responding to a material robot adjacent to the to-be-optimized area based on a position of the to-be-optimized area and a stacking shape of surrounding electronic device materials, and triggering the material robot to adjust positions of the electronic device materials in the to-be-optimized area.

2. The material pile accumulation shape feature extraction method according to claim 1, characterized in that, The application comprises: Collecting storage positions of the electronic device materials, and determining real-time storage images of the electronic device materials based on dynamic shooting of the storage positions by a camera, and determining a background area and a material area according to division of the real-time storage images; In the material area, determining multiple material markers according to traversal of the material area, and determining a shape sketch of each electronic device material based on shape detection of each material marker, and determining self shapes of the electronic device materials according to synchronous recognition of the shape sketches; Determine the current position of each electronic device material based on the position detection of the position mark of each material, and determine the relative position of the plurality of electronic device materials based on the position comparison of the current position of each electronic device material, and construct the corresponding multi-modal data based on the relative position of the plurality of electronic device materials and the self-form.

3. The material pile shape feature extraction method according to claim 1, characterized in that, Determine the actual stacking form of the electronic device material based on the stacking recognition of the real-time storage image, and determine the abnormal stacking form feature of the electronic device material according to the actual stacking form and the theoretical stacking form of the electronic device material, including: Collect real-time storage images, and perform stacking recognition on the real-time storage images. The real-time storage images gradually present each actual stacking contour in the process of stacking recognition, and the actual stacking form of the electronic device material is determined based on the synthesis of each actual stacking contour and the position mark of each electronic device material.

4. The material pile shape feature extraction method according to claim 3, characterized in that, Determine the actual stacking form of the electronic device material based on the stacking recognition of the real-time storage image, and determine the abnormal stacking form feature of the electronic device material according to the actual stacking form and the theoretical stacking form of the electronic device material, and further comprising: Collect the theoretical stacking form of the electronic device material, and mark the corresponding difference part according to the comparison between the theoretical stacking form and the actual stacking form of the electronic device material; Determine the abnormal position of a part of the electronic device material based on the traversal of the difference part, and determine the abnormal stacking form feature of the electronic device material according to the synthesis of the abnormal position of a part of the electronic device material.

5. The material pile shape feature extraction method according to claim 1, characterized in that, The plurality of abnormal stacking positions of the abnormal stacking form feature are marked, and the to-be-optimized area is determined according to the use sequence of the electronic device material corresponding to the plurality of abnormal stacking positions, including: Real-time monitoring of abnormal stacking form features, determining the corresponding abnormal stacking distribution map based on the detection of abnormal stacking form features, marking a plurality of abnormal stacking positions in the abnormal stacking distribution map, and presenting the electronic device material corresponding to the plurality of abnormal stacking positions.

6. The material pile accumulation shape feature extraction method according to claim 5, characterized in that, The plurality of abnormal stacking positions of the abnormal stacking form feature are marked, and the to-be-optimized area is determined according to the use sequence of the electronic device material corresponding to the plurality of abnormal stacking positions, and further comprising: Determine the corresponding use schedule based on the traceability of the electronic device material corresponding to the plurality of abnormal stacking positions, and determine the use sequence of the plurality of electronic device materials according to the matching of the use schedule; at the same time, determine a plurality of posture parameters based on the posture detection of the electronic device material corresponding to the plurality of abnormal stacking positions, and determine the current posture of the electronic device material according to the plurality of posture parameters and the form of the corresponding electronic device material; Determine the first sub-optimization area according to the plurality of abnormal stacking positions and the current posture of each electronic device material, determine the second sub-optimization area according to the use sequence of the electronic device material corresponding to the plurality of abnormal stacking positions, and determine the to-be-optimized area based on the synthesis of the first sub-optimization area and the second sub-optimization area.

7. The material pile shape feature extraction method according to claim 1, characterized in that, The to-be-optimized area is based on the position and the stacking form of the electronic device material in the surrounding area, and the material robot adjacent to the to-be-optimized area is triggered to adjust the position of the electronic device material in the to-be-optimized area, including: The position of the to-be-optimized region is determined based on the position detection of the to-be-optimized region, and the position of the to-be-optimized region is periphery detected to mark electronic device materials at the periphery position of the to-be-optimized region, at this time, the form of the electronic device materials at the periphery position of the to-be-optimized region is detected, and the stacking form of the electronic device materials at the periphery is determined.

8. The material pile shape feature extraction method according to claim 7, characterized in that, The to-be-optimized region is adjacent to the material robot, and the position of the to-be-optimized region is adjusted based on the position of the to-be-optimized region and the stacking form of the electronic device materials at the periphery, and the material robot is triggered to adjust the position of the electronic device materials in the to-be-optimized region. The first stacking optimization path is determined according to the region form of the to-be-optimized region and the stacking form of the electronic device materials at the periphery, and the second stacking optimization path is determined according to the region position of the to-be-optimized region and the stacking form of the electronic device materials at the periphery. The material robot is collected which is adjacent to the to-be-optimized region, and the current position of the material robot is marked, the material optimization path of the material robot to the electronic device materials in the to-be-optimized region is determined according to the synthesis of the current position of the material robot, the first stacking optimization path and the second stacking optimization path, and the material robot is triggered to adjust the position of the electronic device materials in the to-be-optimized region based on the execution of the material optimization path.

9. A material pile morphology feature extraction system, comprising: The material stacking form feature extraction system is applied to the material stacking form feature extraction method in any one of claims 1-8, and the material stacking form feature extraction system comprises: A material recognition module is configured to collect real-time storage images of electronic device materials, and determine relative positions and self-forms of a plurality of electronic device materials based on material recognition of the real-time storage images. A theoretical stacking form module is configured to mark contact positions between a plurality of electronic device materials, and determine stacking events of electronic device materials according to a plurality of contact positions, relative positions and self-forms of a plurality of electronic device materials; and determine theoretical stacking forms of electronic device materials according to matching of the stacking events of electronic device materials and use schedules of each electronic device material. An abnormal stacking form feature module is configured to determine actual stacking forms of electronic device materials based on stacking recognition of real-time storage images, and determine abnormal stacking form features of electronic device materials according to the actual stacking forms of electronic device materials and the theoretical stacking forms. A to-be-optimized region module is configured to mark a plurality of abnormal stacking positions of abnormal stacking form features, and determine a to-be-optimized region according to a plurality of abnormal stacking positions and use sequences of corresponding electronic device materials. A position adjustment module is configured to respond to a material robot adjacent to the to-be-optimized region based on the position of the to-be-optimized region and the stacking form of the electronic device materials at the periphery, and trigger the material robot to adjust the position of the electronic device materials in the to-be-optimized region.

Images