Material information marking method, device and electronic equipment

Abstract

The application relates to the technical field of artificial intelligence, and provides a material information marking method and device and electronic equipment. The method comprises the following steps: determining a plurality of color block label images; each color block label in each color block label image comprises a plurality of color blocks; the plurality of color block label images comprise color block labels with different color combinations; the plurality of color block label images are used for representing the same material information; a classification result of the plurality of color block label images is determined; the plurality of color block label images are input into a classifier to obtain a classification result output by the classifier; a prediction accuracy is calculated based on the classification result and preset labels of the plurality of color block label images; a target color block label image is determined based on the color block label image with the highest prediction accuracy; and the color combination of the color block label in the target color block label image is used for marking the material information. The embodiment of the application is used to solve the technical problem that the method of marking the material information by using simple color block labels, two-dimensional codes and bar codes has poor effect.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method, apparatus, and electronic device for marking material information. Background Technology

[0002] Currently, modern automated storage and retrieval systems (AS / RS) manage material information using simple color-block labels, QR codes, or barcodes. Simple color-block labels, for example, cannot easily identify all material information; they rely on a single color. QR codes and barcodes require precise placement angles and locations, and demand high accuracy and range from the image acquisition equipment, making them unsuitable for modern AS / RS material information management. In other words, existing methods of labeling material information using simple color-block labels, QR codes, and barcodes are ineffective. Summary of the Invention

[0003] This application provides a material information marking method, apparatus, and electronic device to solve the technical problem that simple color block labels, QR codes, and barcodes are not effective for marking material information.

[0004] In a first aspect, embodiments of this application provide a material information marking method, including:

[0005] Multiple color block label images are determined; each color block label image includes multiple color block colors; the multiple color block label images include images of color block labels with different color combinations; the multiple color block label images are used to represent the same material information;

[0006] Determine the classification result of the multiple color block label images: input the multiple color block label images into the classifier to obtain the classification result output by the classifier;

[0007] The prediction accuracy is calculated based on the classification results and the preset labels of the multiple color block label images. The target color block label image is determined based on the color block label image with the highest prediction accuracy. The color combination of the color block labels in the target color block label image is used to mark the material information.

[0008] Secondly, embodiments of this application provide a material information marking device, comprising:

[0009] A color block label determination module is used to determine multiple color block label images; each color block label image includes multiple color block colors; the multiple color block label images include images of color block labels with different color combinations; the multiple color block label images are used to represent the same material information;

[0010] The classification result determination module is used to determine the classification result of the multiple color block label images: inputting the multiple color block label images into the classifier to obtain the classification result output by the classifier;

[0011] The target color block label image determination module is used to calculate the prediction accuracy based on the classification result and the preset labels of the multiple color block label images, and to determine the target color block label image based on the color block label image with the highest prediction accuracy; the color combination of the color block labels in the target color block label image is used to mark the material information.

[0012] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the material information marking method described in the first aspect.

[0013] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the material information marking method described in the first aspect.

[0014] The material information marking method, apparatus, and electronic device provided in this application use a classifier to classify multiple different color block labels representing the same material information, calculate prediction accuracy, and determine the target color block label based on the highest prediction accuracy. Since the target color block label includes multiple color blocks, it can store more information than a simple color block label. Compared to QR codes or text information which require close-range, high-precision photography, the recognition and reading of color blocks on color blocks requires lower image pixel counts, saving on industrial camera costs. Simultaneously, color block labels have lower requirements for shooting angle and ambient light, facilitating manual operation and being more suitable for materials requiring low-light storage. Therefore, the target color block label obtained by using a classifier trained in this application can improve the effect of material information marking. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is one of the flowcharts illustrating the material information marking method provided in the embodiments of this application;

[0017] Figure 2 This is a second schematic flowchart of the material information marking method provided in the embodiments of this application;

[0018] Figure 3 This is a schematic diagram of the structure of the color block label provided in the embodiments of this application;

[0019] Figure 4 This is the third flowchart illustrating the material information marking method provided in the embodiments of this application;

[0020] Figure 5 This is the fourth flowchart illustrating the material information marking method provided in the embodiments of this application;

[0021] Figure 6 This is a schematic diagram of the material information marking device provided in the embodiments of this application;

[0022] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Please refer to Figure 1 This application provides a material information marking method, which may include:

[0025] Step 100: Determine multiple color block label images.

[0026] An electronic device identifies multiple color block label images. Each color block label image contains multiple color block colors; the multiple color block label images include images of color block labels with different color combinations; the multiple color block label images are used to represent the same material information. Multiple color block colors can represent more information than a single color, and two color block colors can be used to mark information bits (0 and 1) in binary. Combinations of two color block colors with multiple data bits can represent a large amount of material information. For example, each color block label can contain a total of 12 color blocks, where the start and end bits are position markers (red and black), and the middle 10 bits are information bits. Three of the middle 10 bits are color block number markers near the black color block, and seven bits are information identifiers near the red color block. According to the binary design, a color block label can be composed of up to eight color block labels, expressing data on the order of 128 to the power of 7, approximately 10^23, which can fully manage the storage information of materials.

[0027] The use of multiple different color-coded labels to represent the same material information is illustrated by the following example: For instance, material information can be represented by 12 binary bits of 110101111001. This material information can be represented by four different color-coded labels: black-white-blue-white-blue-white-white-white-blue-blue-red (black as start, red as end), black-white-green-white-green-white-white-white-green-green-red (black as start, red as end), black-red-blue-red-blue-red-red-red-red-blue-blue-yellow (black as start, yellow as end), and black-white-yellow-white-white-white-white-yellow-green (black as start, green as end).

[0028] It should be noted that multiple color block label images can be obtained by taking pictures of various color block labels under different shooting conditions.

[0029] The shooting conditions included different shooting angles and directions for the color block labels. Shooting angles included shooting from below, shooting from above, and shooting at the same level as the color block labels. Shooting directions included shooting from the left and right sides of the color block labels. Multiple images were obtained by shooting four types of color block labels under different shooting conditions: black-white-blue-white-blue-white-white-white-blue-blue-red (black as the start position, red as the end position), black-white-green-white-green-white-white-white-white-green-green-red (black as the start position, red as the end position), black-red-blue-red-blue-red-red-red-red-red-blue-blue-yellow (black as the start position, yellow as the end position), and black-white-yellow-white-white-white-white-yellow-green (black as the start position, green as the end position).

[0030] It should be noted that 1,000 images of each color block label were taken from different angles and directions, but the image quality (pixels, color block size) of different color block labels was kept basically consistent.

[0031] In addition, each color block label image has a corresponding preset label. For example, the preset label generated for the four color block labels are 110101111001.

[0032] Step 200: Determine the classification result of the multiple color block label images: Input the multiple color block label images into the classifier to obtain the classification result output by the classifier.

[0033] An electronic device inputs multiple images into a classifier and obtains multiple classification results. The classifier in this embodiment can be a deep residual network, a random forest, etc. This embodiment uses a deep residual network as an example.

[0034] Due to variations in lighting and shooting angle, the colors of the captured color blocks often differ from the set colors, making it difficult to determine the color combinations. Therefore, this embodiment uses a deep residual neural network to determine color combinations to achieve the best recognition results.

[0035] When object detection methods are applied to different color combinations, it is difficult to label color patches because the main limitation of supervised object detection is the need for a large amount of manual labeling work, which is both expensive and impractical. In this embodiment, we use a classification method based on deep residual neural networks to select appropriate color combinations. Compared with object detection methods, deep residual neural networks require far less labeling work.

[0036] In this implementation, the deep residual network can adopt the ResNet101 network structure for image classification. The ResNet101 network structure is a popular deep residual network. The ResNet101 network structure first performs a 7×7×64 convolution on the input image, i.e., the first convolutional layer, and then passes it through 4 blocks. Each block has 3, 4, 23, and 3 bottlenecks, so the total number of blocks is 3+4+23+3=33. Each block has three layers, so there are 33×3=99 layers. Finally, there is a fully connected layer (for classification). Unlike general classification models that directly output classification results (e.g., cat or dog), the output of the model in this embodiment is a ten-digit number. Therefore, the preset label is a ten-digit number, and the output of the last layer is ten sigmoid functions.

[0037] In this embodiment, images are classified according to preset labels (without annotation) and corresponding label files are generated; the acquired images are divided into training, validation, and test sets in a 7:2:1 ratio. The images are fed into a ResNet101 network for learning, and the output is a ten-digit number.

[0038] Step 300: Calculate the prediction accuracy based on the classification results and the preset labels of the multiple color block label images, and determine the target color block label image based on the color block label image with the highest prediction accuracy.

[0039] The electronic device calculates the prediction accuracy based on the classification results and preset labels of the multiple color block label images, and determines the target color block label image based on the color block label image with the highest prediction accuracy. The color combination of the color block labels in the target color block label image is used to mark the material information. For different types of color block label images, the prediction accuracy is calculated using the preset labels corresponding to different images. For example, if 50 out of 100 images of black, white, blue, white, blue, white, white, white, blue, blue, red color block labels have a classification result of 110101111001, which is the same as the preset label 110101111001, then the prediction accuracy for the black, white, blue, white, blue, white, white, white, blue, blue, red color block labels is 50%. Similarly, assume the prediction accuracy for black, red, blue, red, blue, red, red, red, red, blue, blue, yellow color block labels is 60%. Assume the prediction accuracy for black, white, green, white, green, white, white, white, white, green, green, red color block labels is 75%. Assume the prediction accuracy for black, white, yellow, white, white, white, white, yellow, yellow, green color block labels is 55%. Therefore, black, white, green, and red are chosen as the target color block labels. The starting bit is black, the ending bit is red, and the information bits are represented by white (1) and green (0).

[0040] As can be seen, the embodiments of this application obtained a color combination (black, white, red, and green color block labels) with a classification accuracy of 75% without any annotation. This experimental result confirms that even without labeling specific color block positions, sizes, or other information, the classification model of the deep residual network in this application can learn the distinguishing features of colors and determine the color combination with the highest prediction accuracy.

[0041] It should be noted that in other embodiments, images of different color block labels for multiple material information can be captured simultaneously, and then classified using a deep residual network. For example, for a given set of four types of material information (such as 110101111001, 100110010101, 110000001011, 111100001011), several different color block combinations can be designed (such as black, white, blue, and red; black, white, green, and red; black, red, blue, and yellow; black, white, yellow, and green; etc.).

[0042] By using a classifier to classify multiple different color block labels representing the same material information and calculating the prediction accuracy, the target color block label is determined based on the highest prediction accuracy. Since the target color block label includes multiple color blocks, it can store more information than a simple color block label. Compared to QR codes or text information, which require close-range, high-precision photography, the recognition and reading of color blocks from color blocks on color blocks requires lower image pixel counts, saving on industrial camera costs. Simultaneously, color block labels have lower requirements for shooting angle and ambient light, facilitating manual operation and being more suitable for materials requiring low-light storage. Therefore, this embodiment of the application improves the effectiveness of material information labeling by using target color block labels obtained through classifier training.

[0043] For other aspects of the embodiments of this application, please refer to Figure 2 Step 300, after calculating the prediction accuracy based on the classification results and the preset labels of the multiple color block label images, and determining the target color block label image based on the color block label image with the highest prediction accuracy, further includes:

[0044] Step 400: Obtain multiple color block label images.

[0045] The electronic device acquires multiple color block label images. Please refer to... Figure 3 The color block labels of the color block label image are set for the material based on the color rules of the color combination of the target color block label. In one embodiment, the color rules of the color combination of the target color block label include a first color block color rule for the start position, a second color block color rule for the end position, and a third color block color rule for the information position; the information position is located between the start position and the end position. For example, the color rules of the color combination of the target color block label include marking the start position with black (…). Figure 3 1), two colors (green and white), Figure 3 (2 and 3) are used to mark binary information bits (0 and 1), red ( Figure 3 4) is used to mark the end position, with each small color block separated by a black box. The information bits can include multiple bits, such as a 10-bit binary number. For example, in one embodiment, 110101111001 is represented by black-white-green-green-white-white-green-white-white-red using color block labels.

[0046] Workers can affix color-block labels to all materials. Specifically, one or more color-block labels can be affixed to the outer surface of the material using adhesive, or created by heat pressing, cold pressing, engraving, or other methods to form recessed or raised color-block labels on the material's outer surface. These color-block labels are used to encode and identify at least the material's type, quantity, date, location, and other storage information. After affixing one or more color-block labels to the outer surface of the material, the material is stored in the corresponding storage space. Multiple color-block label images are obtained by workers taking photos of the material's color-block labels under different shooting environments and conditions; the shooting environment includes lighting conditions and / or the material's background; the shooting conditions include shooting angle and / or shooting distance.

[0047] Traditional barcodes only convey information in one direction (usually horizontally), leaving the vertical direction uninformed. Their height is typically for reader alignment. Changes in angle or lighting significantly impact recognition. Furthermore, damage or obstruction renders barcodes unreadable. This necessitates not only ensuring the barcode's integrity and independence but also requiring close-range handheld reading of the material. QR codes improve upon barcodes, allowing for a 50% damage rate, but still require maintaining a certain distance and angle from the material during scanning, incurring higher manual costs. Additionally, black-and-white QR code recognition is severely affected in low-light environments, limiting their application on many materials (e.g., film, volatile chemicals).

[0048] This application allows for the capture of color block labels on materials under various lighting conditions and from different shooting angles, resulting in multiple images of the color block labels. Lighting conditions include bright sunlight and dimly lit indoor environments. Shooting angles include oblique views, upward views, and downward views.

[0049] This application can also photograph the color block labels of materials under different lighting conditions and at different shooting distances to obtain multiple color block label images. The lighting conditions include bright sunlight and dimly lit indoor environments. The shooting angles include close-up, long-distance, and ultra-long-distance, such as 10cm, 1m, and 10m.

[0050] This application can also photograph the color block labels of materials using different backgrounds and shooting angles to obtain multiple color block label images. The backgrounds include both cluttered and simple backgrounds. Shooting angles include oblique, upward, and downward views.

[0051] This application can also photograph the color block labels of materials using different backgrounds and shooting distances to obtain multiple color block label images. The backgrounds can be cluttered or simple. Shooting angles include close-up, long-distance, and ultra-long-distance shots, such as 10cm, 1m, and 10m.

[0052] In one embodiment, for color block labels, this application embodiment learned from over 2,300 sets of images, totaling approximately 50,000 data points, under various lighting conditions, shooting angles, shooting distances, and shooting backgrounds during the data acquisition phase. These included scenes in severely limited lighting conditions (dark rooms), oblique views (up to 60°), upward views, downward views, and long-distance and ultra-long-distance scenes. Due to the unique characteristics of color block labels, they exhibit a certain degree of stability in response to changes in lighting conditions. Furthermore, due to the diversity of this data, including color block label images from various angles and distances, identification can be performed at various shooting angles and distances. This allows material management personnel to accurately obtain color block information using frame extraction from common fixed-position industrial cameras or webcams.

[0053] Step 500: Identify the color block labels in the color block label image based on the object detection method.

[0054] Electronic devices identify color block labels in color block label images using object detection methods. The object detection method in deep learning is used to identify color blocks. The model, which has been repeatedly trained on a large dataset, has excellent robustness and can identify color block label image information under various lighting and shooting angles. It has low requirements for the lighting, size, and accuracy of the acquired images.

[0055] Step 600: Search for each color block in the color block label image based on the angle threshold between adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks.

[0056] After acquiring the color block label image, the camera in this application transmits the color block information to the terminal for post-processing. This application's embodiments utilize a post-processing method that integrates three types of prior information for sequential color block searching. These three types of prior information are: the angle between color blocks represents the continuity of their positions; the distance between the center points of two adjacent detection boxes represents the size of the color block; and the area ratio of adjacent color blocks. The number of color blocks determines the number of objects to be detected between the start and end positions.

[0057] The angle between color blocks represents the angle between the lines connecting the center points of the color blocks. The number of color blocks determines the number of objects detected between the start and end positions. First, all color block labels are searched in the entire color band image to find a start position color block, which is then marked as the first color block of a color block label. Then, the next color block is determined by the area ratio, distance, and angle relationship between the next color block and the first color block. Experiments have shown that for adjacent color blocks with the same color block label, the distance between the center points of two adjacent detection boxes should be consistent, and the distance between the center points should be within the corresponding threshold. The angle between adjacent detection boxes should also be within 15°, and the area ratio between adjacent detection boxes should also be within the area ratio threshold range. This step is repeated until 10 information bits in the color block label are found. Simultaneously, the end position needs to be marked as the end position color block. At this point, the color block label search is complete. This process is repeated until all color blocks corresponding to all start positions in the image are detected.

[0058] This application employs a patch-by-patch search method in color-block labeled images, based on thresholds for the angle between adjacent detection boxes, the distance between the center points of adjacent detection boxes, the area ratio of adjacent detection boxes, and the number of color blocks. This post-processing method integrates prior information such as the angle, size, distance, and number of adjacent color blocks to control the continuity and consistency of the detection results. Using this information, we filter and optimize a large number of duplicate detection boxes, ensuring a one-to-one correspondence between the predicted boxes and the actual color block labels. This guarantees the accuracy and robustness of detection when multiple color block labels are present in an image.

[0059] In other embodiments of this application, step 600, searching for color block labels in the color block label image one by one based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks, includes:

[0060] Step 610: Search for the start or end color block in the color block label and obtain the detection box of the start or end color block.

[0061] The system searches for start or end color blocks in a color block label using an electronic device, obtaining the detection bounding box for each start or end color block. For example, it can search for a black start color block in a color block label to obtain the detection bounding box for the black color block; or it can search for a red start color block in a color block label to obtain the detection bounding box for the red color block. The system then obtains the length L_0, width W_0, center point coordinates (x_0, y_0), and confidence level p0 of the start or end color block in the color block label.

[0062] Repeat the following steps until the number of color blocks found reaches the required number, or until a color block corresponding to the start position and the stop position is found, or a color block corresponding to the stop position and the start position is found:

[0063] Step 620: Search for the first color block in the color block label, and obtain multiple candidate boxes for the first color block based on the detection box of the first color block.

[0064] Step 630: Determine the target candidate box from multiple candidate boxes based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, and the area ratio threshold range of adjacent detection boxes.

[0065] The first color block is the color block in the color block label that contains all information bits except for the end or start color block, denoted as the Nth color block. Consider the center point coordinates (x_N-1, y_N-1) of the (N-1)th color block. Obtain multiple candidate boxes for the first color block. Search for the center points of candidate boxes within a radius of 0.9K to 1.1K, centered on the center point of the Nth color block. And the candidate boxes are required to satisfy Candidate boxes that do not meet the requirements are removed. Where L N-1 W represents the length of the (N-1)th color block. N-1 L represents the width of the (N-1)th color block. 候选 W represents the length of the candidate box. 候选 This represents the width of the candidate box. The angle between the center points of the Nth color block and the (N-1)th color block is... (x N ,y N (x) represents the coordinates of the center point of the Nth color block. N-1 ,y N-1 Let θ be the coordinates of the center point of the (N-1)th color block. The angle between the Nth color block and the next candidate detection box needs to satisfy |θ|. N -θ 候补 |<15°, and the second derivative of the angle The target candidate box is determined from multiple candidate boxes by using the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, and the area ratio threshold range of adjacent detection boxes.

[0066] Step 640: Determine that a target candidate box exists, use the target candidate box as the detection box of the first color block, and continue the search for the next color block of the target candidate box.

[0067] When a target candidate box exists that satisfies the thresholds for the included angle between adjacent detection boxes, the distance between the center points of adjacent detection boxes, and the area ratio of adjacent detection boxes, the target candidate box is used as the detection box for the first color block. Steps 620-640 are repeated to continue searching for the next color block of the target candidate box. This continues until the number of color blocks in the searched information bits meets the required number of color blocks (e.g., 10), and the last detected stop or start color block is found.

[0068] In other aspects of the embodiments of this application, after step 630, which determines the target candidate box from multiple candidate boxes based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, and the area ratio threshold range of adjacent detection boxes, it further includes:

[0069] Step 650: If it is determined that no target candidate box exists, mark the target candidate box as a missing position; fill the missing position based on the detection box of the first color block, use the detection box of the missing position as the detection box of the first color block, and continue the search for the next color block of the detection box of the missing position.

[0070] If no target candidate box is found, the target candidate box is marked as a missing position (filled with N points). The detection box of the missing position is predicted based on the length, width, and center point of the detection box of the first color block. That is, the missing position is filled based on the detection box of the first color block, and then the missing detection box is used as the detection box of the first color block. Steps 620-640 are repeated to continue searching for the next color block of the missing position detection box. This continues until the number of color blocks of the searched information bits meets the number of color blocks (e.g., 10), and the last detected stop or start position color block is found.

[0071] In this embodiment, all color block labels are searched throughout the entire color block label image to find a start or end color block, which is then marked as the first color block of a color block label. The next color block is then determined based on its area ratio, distance, and angle relationship with the previous color block. For areas that cannot be detected based on prior information, predictions are made using existing detection boxes to fill in an N-point in the area where a color block should appear, indicating a missing position in the color block detection. The search continues based on the positions of the missing positions filled in by the known color block detection boxes until a complete color block label has been searched.

[0072] This application's post-processing method for searching color patches integrates prior information such as the angle, size, distance, and number of adjacent color patches to control the continuity and consistency of the detection results. Using this information, we filter and optimize a large number of duplicate detection boxes, ensuring a one-to-one correspondence between the predicted boxes and the actual color patch labels. This guarantees the accuracy and robustness of detection when multiple color patch labels are present in an image.

[0073] For other aspects of the embodiments of this application, please refer to Figure 4 Step 600, after searching each color block in the color block label image based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks, further includes:

[0074] Step 700: Perform perspective transformation on the color block label image.

[0075] Due to camera angle issues, excessive distortion of the color block label image can occasionally lead to missed detections and misidentifications. By utilizing the three prior pieces of information from step 600 and the continuity of the color blocks, the average tilt angle and distance of the color blocks are obtained. Then, appropriate rotation, affine transformations, and perspective transformations are applied to correct the image, ensuring that the originally tilted color blocks are now horizontal or vertical for detection. Classic object detection networks can only generate positive rectangular bounding boxes, suitable for natural objects like cats and dogs, but ineffective for tilted quadrilaterals. For tilted color blocks, attempting to encompass them with rectangular bounding boxes inevitably introduces irrelevant information (noise), especially noticeable in small color blocks, where over 60% of the area is invalid. This severely interferes with detection. Therefore, we utilize the continuity of the color blocks, calculating a rotation angle based on the tilt angle of the continuous color block labels, and then rotating and correcting the color blocks to ensure that the detection boxes are all positive rectangles, containing as little redundant information (noise) as possible.

[0076] Specifically, when image distortion and warping are severe, ordinary rotation correction methods are no longer sufficient to improve detection performance. This application introduces perspective transformation to address this issue. Perspective transformation projects an image onto a new viewing plane, also known as projection mapping. The purpose of perspective transformation is to convert objects that are straight lines in reality, which may appear as oblique lines in an image, into straight lines. This application applies perspective transformation to color block label images, correcting distorted and warped images of color block labels into rectangular frames, thereby improving the accuracy of color block label detection.

[0077] In other aspects of the embodiments of this application, after step 700, which involves performing perspective transformation on the color block label image, the method further includes:

[0078] Step 800: Determine the missing positions of the color blocks in the color block label image after searching each color block individually.

[0079] Because industrial internet scenarios are highly complex, including situations with insufficient light, dark rooms, steam, and smoke, the quality of the detected color block label images may be unavoidably affected during actual inspection. For example, when the image quality of the color block label image to be detected is poor (tilted, dimly lit, blurry, etc.), the classification probability score of the detected target will be very low. When there are a large number of missing color block targets in the detected color block label image, this application embodiment implements a post-processing method to gradually reduce the threshold. First, the missing positions of the color block search in the color block label image after searching each color block are determined (i.e., the N points marked in step 900).

[0080] For each missing bit, perform the following steps until all missing bits are filled:

[0081] Step 900: Predict the predicted detection box for the missing position based on the adjacent detection boxes of the missing position.

[0082] The electronic device predicts the predicted detection box of the missing position based on the adjacent detection boxes. In other words, it fills in the predicted detection box of the missing position based on the adjacent detection boxes. For example, if there is a detection box to the left of a missing position in a color block label, the predicted detection box of the missing position is filled based on that detection box. By predicting the predicted detection box of the missing position based on the adjacent detection boxes, the areas of adjacent color blocks in the image are made similar; the angle changes of consecutive adjacent color blocks are consistent, mathematically reflected as the second derivative of the angle being close to 0; and the distance between adjacent color blocks is consistent.

[0083] Step 1000: Reduce the confidence threshold based on the preset step size.

[0084] In this application, the confidence threshold is used to determine whether an object within the detection box is a positive or negative sample. Objects with a confidence threshold greater than the threshold are considered positive samples, while those less than the threshold are considered negative samples (i.e., background). In reality, when the image quality of the color block label to be captured is poor (due to tilting, dim lighting, blurring, etc.), the confidence threshold needs to be lowered to ensure accurate identification of the color blocks in the label. The electronic device lowers the confidence threshold based on a preset step size. For example, if the preset step size is 0.1 and the initial confidence threshold is 0.8, the electronic device will lower the confidence threshold by 0.1 to 0.7.

[0085] Step 1100: Determine the color blocks that meet the confidence threshold after the threshold reduction based on the predicted detection boxes, and use the predicted detection boxes as the detection boxes of the color blocks.

[0086] The electronic device determines that a color block has been detected based on the predicted detection box, which meets the confidence threshold after the threshold has been lowered. This indicates that after lowering the confidence threshold, a color block has been detected within the predicted detection box, and the predicted detection box is used as the detection box for the color block. Steps 900 to 1100 are then repeated until all missing positions are filled.

[0087] For other aspects of the embodiments of this application, please refer to Figure 4 After step 1000, which involves reducing the confidence threshold based on a preset step size, the process also includes:

[0088] Step 1200: Determine that no color block matching the reduced confidence threshold is detected based on the predicted detection box, and continue to execute the step of reducing the confidence threshold based on the preset step size until a color block matching the reduced confidence threshold is detected based on the predicted detection box, and use the predicted detection box as the detection box of the color block.

[0089] If the electronic device determines that no color block matching the reduced confidence threshold is detected based on the predicted detection box, it means that the current reduced confidence threshold still cannot meet the requirements and the color block cannot be detected. It is necessary to continue to execute the step of reducing the confidence threshold based on the preset step size until a color block matching the reduced confidence threshold is detected based on the predicted detection box, and then the predicted detection box is used as the detection box for the color block.

[0090] For example, when the confidence threshold is reduced to 0.7, no color patch with a confidence threshold of 0.7 is detected within the predicted detection box. In this case, the confidence threshold is further reduced to 0.6. It is then determined whether a color patch meeting the confidence threshold of 0.6 is detected based on the predicted detection box. If no color patch meeting the reduced confidence threshold is detected based on the predicted detection box, the confidence threshold is further reduced until a color patch meeting the reduced confidence threshold is detected based on the predicted detection box, or the confidence threshold is reduced to 0.1. The predicted detection box is then used as the detection box for the color patch.

[0091] In this embodiment, the threshold of the predicted detection box with missing positions is reduced locally by a fixed step size. If a color block that meets the current threshold standard is detected, it is recorded and the process returns to step 900. If no color block is detected within the predicted detection box, the confidence threshold is reduced until a color block is detected or the confidence threshold reaches 0.1. Compared to global threshold reduction, the detection accuracy of the detection box is severely affected by noise. This embodiment addresses the problem of noise severely affecting the detection accuracy of the detection box when global threshold reduction is used by reducing the threshold locally.

[0092] It should be noted that if the color block labels (including the 12 bits of the start and end positions) still have missing values, then the confidence threshold in step 1000 is replaced with the angle threshold between adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, and the area ratio threshold range of adjacent detection boxes; steps 900-1200 are repeated until there are no missing values ​​in the color block labels (including the 12 bits of the start and end positions).

[0093] This application detects and identifies color blocks near missing positions in color block labels by locally lowering the threshold, thereby achieving the identification of all color blocks in the color block label image.

[0094] In other aspects of the embodiments of this application, the method of the embodiments of this application further includes:

[0095] Step 1300: Search for each color block in the color block label image based on the angle threshold between adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks.

[0096] Step 1400: Ensure that all color blocks in the color block label image are completely identified, and then transmit the color block label image to the server.

[0097] Please refer to Figure 5 This application performs a block-by-block search on the color block label image after steps 600 (block-by-block search), 700 (perspective transformation correction), and 800-1200 (predictive detection box estimation using a progressively lowering threshold method). Then, it performs a block-by-block search on the color block labels in the image based on thresholds for the included angle between adjacent detection boxes, the distance between the center points of adjacent detection boxes, the area ratio of adjacent detection boxes, and the number of color blocks. Specifically, step 900 is re-executed to ensure complete identification of all color blocks in the color block label image, and the image is then transmitted to the server. The server decodes and re-encodes the color block labels in the image to obtain the material's storage information and performs editing and updates in the background.

[0098] Traditional image processing methods employ segmentation, using various rectangular frames (such as lens capture frames and color wheel label capture frames) of different sizes to traverse the segmented image and determine the positions of the color wheel labels before decoding them. This approach requires numerous manual parameter settings, cannot automatically adjust to environmental changes, and has poor versatility. Furthermore, while simple color block labels are easy to identify, they lack sufficient information. QR codes and barcodes, on the other hand, have high requirements for the angle and position of the materials, and demand high precision and distance from the image acquisition equipment, making them unsuitable for material information management in automated shelving systems.

[0099] This application offers convenient color block label recognition. Compared to QR codes or text information, which require close-range, high-precision photography, this application requires lower image pixel counts for color block label recognition and reading, saving on industrial camera costs. It also has lower requirements for shooting angles and ambient light, facilitating manual operation and making it more suitable for materials requiring low-light storage (such as darkroom warehouses storing film). Secondly, the color block labels of this application are space-saving, occupying minimal space and can be directly affixed to the material surface. Furthermore, the color block labels are not easily detached, as they are directly adhered to the material surface using adhesive, minimizing impact during handling and providing a stable hold. This application fully automates material information management through color block labels. For materials on high-rise shelves in automated warehouses, information collection and recognition are difficult for manual labor. This application, however, can collect all material information for a specific area using a fixed camera position. For distorted or warped images, image correction and information reading are fully automated, saving server-side computing resources.

[0100] This application embodiment uses a target detection method to obtain color block information from images captured by a camera or webcam. It eliminates the need for setting capture frames of different sizes and image segmentation. The detection is less affected by lighting, shooting angle, and shooting distance. The labels use binary color blocks to store information, resulting in a large storage capacity and eliminating the need for high-precision acquisition equipment. Through the color block search step in step 600, the perspective transformation correction in step 700, and the progressive threshold reduction method in steps 800-1200, this application embodiment can adaptively identify the color block information of the color block labels based on environmental changes, achieving full automation without the need for parameter settings.

[0101] The material information marking device provided in the embodiments of this application is described below. The material information marking device described below can be referred to in correspondence with the material information marking method described above.

[0102] Please refer to Figure 6 This application also provides a material information marking device, including:

[0103] The color block label determination module 201 is used to determine multiple color block label images; each color block label image includes multiple color block colors; the multiple color block label images include images of color block labels with different color combinations; the multiple color block label images are used to represent the same material information;

[0104] The classification result determination module 202 is used to determine the classification result of the multiple color block label images: inputting the multiple color block label images into the classifier to obtain the classification result output by the classifier;

[0105] The target color block label image determination module 203 is used to calculate the prediction accuracy based on the classification result and the preset labels of the multiple color block label images, and to determine the target color block label image based on the color block label image with the highest prediction accuracy; the color combination of the color block labels in the target color block label image is used to mark the material information.

[0106] The classification result determination module uses a classifier to classify multiple different color block labels representing the same material information and calculates the prediction accuracy. The target color block label is determined based on the highest prediction accuracy. Since the target color block label includes multiple color blocks, it can store more information than a simple color block label. Compared to QR codes or text information, which require close-range, high-precision photography, the recognition and reading of color blocks from color blocks on color blocks requires lower image pixel counts, saving on industrial camera costs. Simultaneously, color block labels have lower requirements for shooting angle and ambient light, facilitating manual operation and being more suitable for materials requiring low-light storage. Therefore, this embodiment of the application improves the effectiveness of material information labeling by using target color block labels obtained through classifier training.

[0107] In one embodiment, the material information marking device further includes:

[0108] A color block label image acquisition module is used to acquire multiple color block label images. These multiple color block label images are obtained by capturing images of the color block labels on the material under different shooting environments and conditions. The shooting environment includes lighting conditions and / or the material's shooting background. The shooting conditions include shooting angle and / or shooting distance. The color block labels in the color block label images are set on the material based on color rules for the color combinations of the target color block labels. The color rules for the color combinations of the target color block labels include a first color block color rule for the start position, a second color block color rule for the end position, and a third color block color rule for the information position. The information position is located between the start position and the end position.

[0109] The object detection module is used to identify color block labels in color block label images based on object detection methods;

[0110] The color block search module is used to search for color block labels in the color block label image one by one based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks.

[0111] In one embodiment, the color block search includes:

[0112] The first search module is used to search for color blocks at the start or end position of the color block label and obtain the detection box of the color block at the start or end position.

[0113] The repeated search module is used to repeatedly execute the following steps until the number of searched color blocks reaches the required number of color blocks or the color block corresponding to the start position or the color block corresponding to the start position is found: search for the first color block in the color block label, obtain multiple candidate boxes for the first color block based on the detection box of the first color block; determine the target candidate box from the multiple candidate boxes based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, and the area ratio threshold range of adjacent detection boxes; if the existence of the target candidate box is determined, use the target candidate box as the detection box of the first color block, and continue the search for the next color block of the target candidate box.

[0114] In one embodiment, the material information marking device further includes:

[0115] The missing position filling module is used to determine if there is no target candidate box, and then mark the target candidate box as a missing position; based on the detection box of the first color block, the missing position is filled, and the detection box of the missing position is used as the detection box of the first color block, and the search for the next color block of the detection box of the missing position continues.

[0116] In one embodiment, the material information marking device further includes:

[0117] The perspective transformation module is used to perform perspective transformation on color block label images.

[0118] In one embodiment, the material information marking device further includes:

[0119] The missing position determination module is used to determine the missing positions of color blocks in the color block label image after searching each color block;

[0120] The loop execution module is used to perform the following steps for each missing position until all missing positions are filled: predict the predicted detection box of the missing position based on the adjacent detection boxes of the missing position; reduce the confidence threshold based on a preset step size; determine the color block that is detected based on the predicted detection box and meets the confidence threshold after the threshold reduction, and use the predicted detection box as the detection box of the color block.

[0121] In one embodiment, the material information marking device further includes:

[0122] The progressively lowering threshold module is used to determine if a color block that does not meet the confidence threshold after threshold reduction is detected based on the predicted detection box. If so, the step of lowering the confidence threshold based on a preset step size is continued until a color block that meets the confidence threshold after threshold reduction is detected based on the predicted detection box. The predicted detection box is then used as the detection box for the color block.

[0123] In one embodiment, the material information marking device further includes:

[0124] The repeating color block search module is used to search for color block labels in the color block label image one by one based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks.

[0125] The color block label image transmission module is used to ensure complete recognition of all color blocks in the color block label image and transmit the color block label image to the server.

[0126] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call a computer program in the memory 730 to execute the steps of a material information labeling method, such as: determining multiple color block label images; each color block label image includes multiple color block colors; the multiple color block label images include images of color block labels with different color combinations; the multiple color block label images are used to represent the same material information; determining the classification result of the multiple color block label images: inputting the multiple color block label images to a classifier to obtain the classification result output by the classifier; calculating the prediction accuracy based on the classification result and preset labels of the multiple color block label images; determining the target color block label image based on the color block label image with the highest prediction accuracy; the color combination of the color block labels in the target color block label image is used to label the material information.

[0127] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the material information marking method provided in the above embodiments, such as: determining multiple color block label images; each color block label image includes multiple color block colors in its color block label; the multiple color block label images include images of color block labels with different color combinations; the multiple color block label images are used to represent the same material information; determining the classification result of the multiple color block label images: inputting the multiple color block label images into a classifier to obtain the classification result output by the classifier; calculating the prediction accuracy based on the classification result and the preset labels of the multiple color block label images; determining the target color block label image based on the color block label image with the highest prediction accuracy; the color combination of the color block labels in the target color block label image is used to mark the material information.

[0129] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to execute the steps of the material information marking method provided in the above embodiments, such as: determining multiple color block label images; each color block label image includes multiple color block colors in its color block label; the multiple color block label images include images of color block labels with different color combinations; the multiple color block label images are used to represent the same material information; determining the classification result of the multiple color block label images: inputting the multiple color block label images into a classifier to obtain the classification result output by the classifier; calculating the prediction accuracy based on the classification result and preset labels of the multiple color block label images; determining a target color block label image based on the color block label image with the highest prediction accuracy; the color combination of the color block labels in the target color block label image is used to mark the material information.

[0130] Processor-readable storage media can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for marking material information, characterized in that, include: Multiple color block label images are determined; each color block label image includes multiple color block colors; the multiple color block label images include images of color block labels with different color combinations; the multiple color block label images are obtained by photographing the color block labels with different color combinations under different shooting conditions; the multiple color block label images are used to represent the same material information; the color rules for the different color combinations include a first color block color rule for the start position, a second color block color rule for the end position, and a third color block color rule for the information position; the information position is located between the start position and the end position. Determine the classification result of the multiple color block label images: input the multiple color block label images into the classifier to obtain the classification result output by the classifier; The classifier is a deep residual network; The prediction accuracy is calculated based on the classification result and the preset labels of the multiple color block label images. The target color block label image is determined based on the color block label image with the highest prediction accuracy. The color combination of the color block labels in the target color block label image is used to mark material information. The classification result and the preset label are binary information with the same number of bits, and the preset labels of the multiple color block label images are the same. After determining the target color block label image based on the color block label image with the highest prediction accuracy, the process further includes: Multiple color block label images are acquired; these multiple color block label images are obtained by photographing the color block labels of the material under different shooting environments and conditions, wherein the shooting environment includes lighting conditions and / or the shooting background of the material; the shooting conditions include shooting angle and / or shooting distance; the color block labels in the color block label images are set on the material based on the color rules of the color combination of the target color block label; the color rules of the color combination of the target color block label include a first color block color rule for the start position, a second color block color rule for the end position, and a third color block color rule for the information position; the information position is located between the start position and the end position; Identify color block labels in color block label images based on object detection methods; The color block labels in the color block label image are searched one by one based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks.

2. The material information marking method according to claim 1, characterized in that, The step of searching for color blocks in the color block label image one by one based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks includes: Search for the start or end position of the color block in the color block label to obtain the start position. Or the detection frame of the color block at the termination position; Repeat the following steps until the number of color blocks found reaches the specified number, or until a color block corresponding to the start position or the start position corresponding to the stop position is found: Search for the first color block in the color block label, and obtain multiple candidate boxes for the first color block based on the detection box of the first color block; The target candidate box is determined from the plurality of candidate boxes based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, and the area ratio threshold range of adjacent detection boxes. Once the existence of the target candidate box is confirmed, the target candidate box is used as the detection box of the first color block, and the search for the next color block of the target candidate box continues.

3. The material information marking method according to claim 2, characterized in that, After determining the target candidate box from the plurality of candidate boxes based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, and the area ratio threshold range of adjacent detection boxes, the method further includes: If the target candidate box is determined not to exist, the target candidate box is marked as a missing position; the missing position is filled based on the detection box of the first color block, and the detection box of the missing position is used as the detection box of the first color block to continue the search for the next color block of the detection box of the missing position.

4. The material information marking method according to claim 1, characterized in that, The process of searching for each color block in the color block label image based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks further includes: Perform perspective transformation on the color block label image.

5. The material information marking method according to claim 4, characterized in that, After performing perspective transformation on the color block label image, the process also includes: Identify the missing positions in the color patch label image after searching each color patch individually; For each of the missing bits, perform the following steps until all the missing bits are filled: Predict the predicted detection box for the missing position based on the adjacent detection boxes of the missing position. The confidence threshold is reduced based on a preset step size; If a color patch is detected based on the predicted detection box and meets the confidence threshold after the threshold is reduced, the predicted detection box is used as the detection box for the color patch.

6. The material information marking method according to claim 5, characterized in that, After reducing the confidence threshold based on a preset step size, the method further includes: If it is determined that no color block matching the reduced confidence threshold is detected based on the predicted detection box, the step of reducing the confidence threshold based on a preset step size continues until a color block matching the reduced confidence threshold is detected based on the predicted detection box, and the predicted detection box is used as the detection box for the color block.

7. The material information marking method according to claim 1, characterized in that, The method further includes: The color block labels in the color block label image are searched one by one based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks; Once it is confirmed that all color blocks in the color block label image have been completely identified, the color block label image is transmitted to the server.

8. A material information marking device, characterized in that, include: A color block label determination module is used to determine multiple color block label images; each color block label image includes multiple color block colors; the multiple color block label images include images of color block labels with different color combinations; the multiple color block label images are obtained by taking pictures of the color block labels with different color combinations under different shooting conditions; the multiple color block label images are used to represent the same material information; the color rules for the different color combinations include a first color block color rule for the start position, a second color block color rule for the end position, and a third color block color rule for the information position; the information position is located between the start position and the end position. The classification result determination module is used to determine the classification result of the multiple color block label images: inputting the multiple color block label images into the classifier to obtain the classification result output by the classifier; The classifier is a deep residual network; The target color block label image determination module is used to calculate the prediction accuracy based on the classification result and the preset labels of the multiple color block label images, and to determine the target color block label image based on the color block label image with the highest prediction accuracy; the color combination of the color block label in the target color block label image is used to mark the material information; the classification result and the preset label are binary information with the same number of bits, and the preset labels of the multiple color block label images are the same; The material information marking device is used to perform the following steps after the step of determining the target color block label image based on the color block label image with the highest prediction accuracy is completed: Multiple color block label images are acquired; these multiple color block label images are obtained by photographing the color block labels of the material under different shooting environments and conditions, wherein the shooting environment includes lighting conditions and / or the shooting background of the material; the shooting conditions include shooting angle and / or shooting distance; the color block labels in the color block label images are set on the material based on the color rules of the color combination of the target color block label; the color rules of the color combination of the target color block label include a first color block color rule for the start position, a second color block color rule for the end position, and a third color block color rule for the information position; the information position is located between the start position and the end position; Identify color block labels in color block label images based on object detection methods; The color block labels in the color block label image are searched one by one based on the included angle threshold of adjacent detection boxes, the distance threshold between the center points of adjacent detection boxes, the area ratio threshold range of adjacent detection boxes, and the number of color blocks.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the material information marking method according to any one of claims 1 to 7.

Images