Sole gluing area point cloud generation method based on rgbd camouflage instance segmentation

Abstract

This invention discloses a method for generating point clouds of glued areas on shoe soles based on RGBD camouflage instance segmentation. The method includes acquiring a target shoe sole image; constructing a four-modal tensor matrix corresponding to each pixel in the target shoe sole image; inputting the four-modal tensor matrix into a pre-trained shoe sole segmentation model, and generating binary classification probabilities for each pixel through forward propagation; determining J target pixels based on the binary classification probabilities of each pixel; and generating a point cloud of the glued area of ​​the target shoe sole based on the J target pixels. This invention generates an orthogonalized pixel mesh with directional consistency and structural symmetry, thereby ensuring that the spatial relationships around each pixel are accurately captured. Simultaneously, based on pixel-level segmentation, only pixels determined to be glued areas are subjected to 3D reconstruction, ensuring that the point cloud covers the actual glued area and avoiding the introduction of non-target area points.

Description

Technical Field

[0001] This invention relates to a method for generating glued point clouds, specifically a method for generating point clouds of glued areas on shoe soles based on RGBD camouflage instance segmentation. Background Technology

[0002] In the shoe sole manufacturing process, automated glue application systems rely on precise three-dimensional path information to achieve the process requirements of uniform glue lines and clear boundaries. Existing technologies typically use RGBD cameras to acquire color and depth images of the shoe sole, first performing back-projection processing on the entire depth image to generate a complete three-dimensional point cloud of the shoe sole, and then extracting the outline of the glue application area or fitting the glue application path based on this point cloud. Document CN110226806B discloses a method and apparatus for generating shoe sole glue application trajectories, which can solve the problems of inaccurate extraction of shoe sole edges or poor glue application effects and the easy generation of defective products when using CAD models for shoe sole spraying in existing technologies.

[0003] However, the existing technologies disclosed in the aforementioned literature acquire a relatively complete geometric model during the point cloud acquisition process, but they have obvious shortcomings in practical applications:

[0004] First, the generated point cloud contains information about the entire surface of the sole, resulting in a large data volume and high redundancy. The area actually used for adhesive application only occupies a small portion of the total surface area, leading to a heavy computational burden and slow response time. Second, in scenarios with similar textures, uneven lighting, or visual camouflage (such as blurred boundaries between dark glue lines and dark soles), traditional segmentation methods struggle to accurately extract the adhesive application area. If the entire image is still back-projected, a large number of erroneous or uncertain areas will be introduced into the 3D space, causing distortion. Therefore, this invention provides a method for generating point clouds of the adhesive application area on soles based on RGBD camouflage instance segmentation. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for generating point clouds of adhesive-coated areas on shoe soles based on RGBD camouflage instance segmentation. This method solves the technical problems raised in the background by first segmenting target pixels and then back-projecting the target pixels to generate point clouds of adhesive-coated areas.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for generating point clouds of the glued area on the sole based on RGBD camouflage instance segmentation includes:

[0008] S1. Obtain the target shoe sole image; wherein, the target shoe sole image contains the RGB features and depth features of each pixel;

[0009] S2. Construct the four-modal tensor matrix corresponding to each pixel on the target shoe sole image;

[0010] The shape of the four-modal tensor matrix is ​​H×W, where H and W are the number of pixels in the height and width directions, respectively.

[0011] S3. Input the four-modal tensor matrix into the pre-trained shoe sole segmentation model, and generate the binary classification probability of each pixel through forward propagation;

[0012] S4. Determine J target pixels based on the binary classification probability of each pixel;

[0013] S5. Generate a point cloud of the glued area of ​​the target shoe sole based on J target pixels.

[0014] In some specific embodiments, constructing the four-modal tensor matrix corresponding to each pixel on the target shoe sole image includes:

[0015] S2-1. Construct UV coordinates for each pixel on the target shoe sole image;

[0016] S2-2. Based on the UV coordinates constructed for each pixel, define a pixel grid for each pixel;

[0017] S2-3. Extract the RGB features and depth features corresponding to the pixel grid on the target shoe sole image;

[0018] S2-4. Concatenate the RGB features and depth features located in the same pixel grid to generate a four-modal tensor corresponding to the pixel.

[0019] S2-5. Traverse each pixel of the target shoe sole image and repeat S2-1 to S2-4 until H×W pixels are generated as four-modal tensors.

[0020] S2-6. Arrange the H×W four-modal tensors according to their spatial positions based on UV coordinates to construct a four-modal tensor matrix.

[0021] In some specific embodiments, the UV coordinates constructed based on each pixel, defining a pixel grid for each pixel, includes:

[0022] S2-2-1, Anchor the UV coordinates of any pixel as the center coordinates;

[0023] S2-2-2. Taking the center coordinates as the starting point, determine the coordinates of its four neighboring regions in the orthogonal direction;

[0024] S2-2-3. Connect the coordinates of the four neighboring domains with the center coordinates to form four spatial lines;

[0025] S2-2-4. Locate the four midpoints of the four spatial lines;

[0026] S2-2-5. Connect the four midpoints in a clockwise or counterclockwise order to form an initial grid centered on the center coordinates;

[0027] S2-2-6. Rotate the initial grid 90 degrees around the center coordinates to generate an orthogonal pixel grid;

[0028] S2-2-7. Traverse the UV coordinates of each pixel until the pixel grid of each pixel is defined.

[0029] In some specific embodiments, the pre-training step of the sole segmentation model includes:

[0030] A1. Obtain the set of camouflaged segmented images;

[0031] A2. Input the camouflaged segmentation image set into the convolutional neural network model for iterative supervised training to generate the shoe sole segmentation model.

[0032] In some specific embodiments, obtaining the camouflaged segmentation image set includes:

[0033] A1-1. Obtain RGBD images of several shoe sole samples;

[0034] A1-2. Perform binary classification labeling on each pixel point in the RGBD images of several shoe sole samples; the binary classification label includes: points to be coated with glue and points not coated with glue;

[0035] A1-3. Perform image enhancement on the RGBD images of several shoe sole samples labeled with binary classification tags to generate several RGBD enhanced images;

[0036] A1-4. Mix the RGBD images of several shoe sole samples labeled with binary classification with several RGBD enhanced images to generate a dummy segmentation image set.

[0037] In some specific embodiments, generating the sole segmentation model includes:

[0038] A2-1. Input the camouflaged segmented image set into the convolutional neural network model;

[0039] A2-2. Extract the RGBD images of the current batch from the camouflaged segmented image set, and generate the pixel prediction mask map of the current batch through forward propagation.

[0040] A2-3. Calculate the cross-entropy loss between the pixel prediction mask of the current batch and the corresponding binary classification labels of the RGBD image;

[0041] A2-4. With the goal of minimizing the cross-entropy loss, update the parameters of the convolutional neural network model, iteratively calculate the cross-entropy loss of the next batch, until the convolutional neural network model converges to the shoe sole segmentation model.

[0042] In some specific embodiments, generating the binary classification probability of each pixel through forward propagation includes:

[0043] S3-1. Using UV coordinates as indices, anchor the target pixel in the four-modal tensor matrix;

[0044] S3-2. Based on K pre-trained convolutional kernels, perform convolution extraction on the four-modal tensors of the target pixel and its neighboring pixels to generate the convolutional feature vector of the target pixel.

[0045] S3-3. Decode the convolutional feature vector of the target pixel to obtain the binary classification probability that the target pixel belongs to the glue-coated area;

[0046] S3-4. Traverse the UV coordinates until the binary classification probability of each pixel in the four-modal tensor matrix is ​​generated.

[0047] In some specific embodiments, determining J target pixels based on the binary classification probability of each pixel includes:

[0048] S4-1. Compare the binary classification probability of each pixel in the four-modal tensor matrix with a set threshold.

[0049] S4-2. If the binary classification probability is not less than the set threshold, then mark the pixel as the point to be coated with glue; otherwise, mark it as the point not to be coated with glue.

[0050] S4-3. Traverse the four-modal tensor matrix and repeat S4-1 to S4-2 until you obtain the pixel mask map corresponding to H×W pixels.

[0051] S4-4. In the pixel mask image, the pixel corresponding to the point to be coated with glue is marked as the target pixel.

[0052] S4-5. Traverse the pixel mask until J target pixels are determined.

[0053] In some specific embodiments, generating a point cloud of the adhesive coating area of ​​the target shoe sole based on J target pixels includes:

[0054] S5-1. Extract the UV coordinates of the target pixel and determine the pixel grid it belongs to based on the UV coordinates;

[0055] S5-2. Extract the depth features of the target pixel based on the pixel grid where the target pixel is located;

[0056] S5-3. Back-project the UV coordinates and depth features of the target pixel to calculate the point cloud coordinates of the target pixel;

[0057] S5-4. Traverse the J target pixels and repeat S5-1 to S5-3 until the glue application area point cloud composed of J point cloud coordinates is generated.

[0058] This invention provides a method for generating point clouds of adhesive-coated areas on shoe soles based on RGBD camouflage instance segmentation, which has the following beneficial effects:

[0059] This invention determines the center coordinates and the coordinates of its four neighboring areas, connects these coordinates to form spatial lines, further locates the midpoints of these lines to construct an initial grid, and then rotates to generate an orthogonal pixel grid with consistent direction and structural symmetry, thereby ensuring that the spatial relationships around each pixel are accurately captured. Furthermore, this invention extracts the UV coordinates of the target pixels and combines them with their corresponding depth features, uses the camera intrinsic parameter matrix for back projection, calculates the 3D coordinates of each target pixel in the camera coordinate system, and generates a point cloud of the adhesive coating area. Based on pixel-level segmentation, only pixels determined to be in the adhesive coating area are subjected to 3D reconstruction, ensuring that the point cloud covers the actual adhesive coating area and avoiding the introduction of non-target area points. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the method for generating point clouds of the adhesive coating area on the sole of a shoe based on RGBD camouflage instance segmentation according to the present invention.

[0061] Figure 2 This is a schematic diagram illustrating the construction process of the four-modal tensor matrix described in this invention;

[0062] Figure 3 This is a schematic diagram illustrating the definition process of the pixel grid described in this invention;

[0063] Figure 4 This is a schematic diagram of the process for determining the target pixel as described in this invention;

[0064] Figure 5 This is a schematic diagram of the point cloud generation process for the adhesive coating area described in this invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Example 1: Please refer to Figures 1 to 5This invention provides a method for generating point clouds of adhesive-coated areas on shoe soles based on RGBD camouflage instance segmentation, comprising the following steps:

[0067] S1. Obtain the target shoe sole image; wherein, the target shoe sole image contains the RGB features and depth features of each pixel;

[0068] Specifically, the target sole image is an RGBD image of the target sole captured by an RGBD camera. That is, for any pixel in the target sole image, in addition to recording the intensity features of the red (R), green (G), and blue (B) channels, the actual distance of the pixel relative to the camera, i.e., the depth feature, is also recorded.

[0069] S2. Construct the four-modal tensor matrix corresponding to each pixel on the target shoe sole image;

[0070] The shape of the four-modal tensor matrix is ​​H×W, where H and W are the number of pixels in the height and width directions, respectively.

[0071] S3. Input the four-modal tensor matrix into the pre-trained shoe sole segmentation model, and generate the binary classification probability of each pixel through forward propagation;

[0072] S4. Determine J target pixels based on the binary classification probability of each pixel;

[0073] S5. Generate a point cloud of the glued area of ​​the target shoe sole based on J target pixels.

[0074] In this embodiment, the method for generating point clouds of the glued area on the sole based on RGBD camouflage instance segmentation acquires a target sole image containing RGB and depth features, constructs a four-modal tensor matrix corresponding to each pixel, and inputs it into a pre-trained sole segmentation model to generate pixel-level binary classification probabilities. After selecting J target pixels according to a set threshold, the three-dimensional point cloud coordinates are calculated, and finally, the corresponding glued area point cloud is generated. This embodiment is based on pixel-level classification results and only performs three-dimensional reconstruction on pixels determined to be glued areas, ensuring that the generated glued area point cloud is consistent with the actual glued area of ​​the target sole.

[0075] Meanwhile, since the RGB features and depth features in the four-modal tensor maintain spatial alignment at the pixel level, the generated point cloud coordinates can realistically reproduce the geometric distribution of the glued area in three-dimensional space, ensuring the geometric shape of the point cloud in the glued area. This provides a clear geometric coordinate input for glue application based on the shoe sole point cloud, improving the efficiency of automated glue application.

[0076] For example, step S2 of the point cloud generation method specifically includes:

[0077] S2-1. Construct UV coordinates for each pixel on the target shoe sole image;

[0078] Specifically, UV coordinates include U coordinates and V coordinates; where the U coordinate represents the position of a pixel in the width direction of the image, that is, the horizontal distance from the left edge of the image to the corresponding pixel. The V coordinate represents the position of a pixel in the height direction of the image, that is, the vertical distance from the top edge of the image to the corresponding pixel.

[0079] Therefore, the UV coordinates represent the position of each pixel in the two-dimensional image coordinate system. The U coordinate represents the horizontal position (column number) of the pixel, and the V coordinate represents the vertical position (row number). The UV coordinates uniquely determine the position of each pixel in the target shoe sole image.

[0080] S2-2. Based on the UV coordinates constructed for each pixel, define a pixel grid for each pixel;

[0081] S2-3. Extract the RGB features and depth features corresponding to the pixel grid on the target shoe sole image;

[0082] Specifically, for each pixel grid, the RGB feature represents the intensity values ​​of the red, green, and blue channels obtained from the RGB data of the target shoe sole image. These values ​​represent the intensity of the corresponding pixel in the red, green, and blue spectrum, respectively, with each color typically ranging from 0 to 255. Simultaneously, based on the pixel grid, the depth of the corresponding pixel is obtained from the depth data of the target shoe sole image, i.e., the actual distance of the corresponding pixel relative to the camera.

[0083] S2-4. Concatenate the RGB features and depth features located in the same pixel grid to generate a four-modal tensor corresponding to the pixel.

[0084] S2-5. Traverse each pixel of the target shoe sole image and repeat S2-1 to S2-4 until H×W pixels are generated as four-modal tensors.

[0085] S2-6. Arrange the H×W four-modal tensors according to their spatial positions based on UV coordinates to construct a four-modal tensor matrix.

[0086] In this embodiment, by establishing UV coordinates for each pixel and constructing a pixel grid, a structured representation of the spatial location of the image is achieved. Based on this, the RGB features and depth features of each pixel are extracted simultaneously and concatenated into a four-modal tensor. Finally, a complete four-modal tensor matrix is ​​constructed according to the spatial arrangement relationship of the UV coordinates.

[0087] This matrix aligns and stitches RGB features (color information) and corresponding depth features at the pixel level, and organizes them in an orderly manner according to their spatial position in the image, forming a multimodal image data structure with spatial consistency, which is suitable for the segmentation of the shoe sole coating area described in this embodiment.

[0088] Furthermore, step S2-2 also includes:

[0089] S2-2-1, Anchor the UV coordinates of any pixel as the center coordinates;

[0090] S2-2-2. Taking the center coordinates as the starting point, determine the coordinates of its four neighboring regions in the orthogonal direction;

[0091] Specifically, the four neighboring coordinates represent the UV coordinates of the four neighboring pixels in the orthogonal directions (horizontal and vertical directions) of the center coordinates; while the neighboring pixels represent the directly adjacent pixels above, below, to the left and to the right of any given pixel.

[0092] S2-2-3. Connect the coordinates of the four neighboring domains with the center coordinates to form four spatial lines;

[0093] S2-2-4. Locate the four midpoints of the four spatial lines;

[0094] S2-2-5. Connect the four midpoints in a clockwise or counterclockwise order to form an initial grid centered on the center coordinates;

[0095] S2-2-6. Rotate the initial grid 90 degrees around the center coordinates to generate an orthogonal pixel grid;

[0096] S2-2-7. Traverse the UV coordinates of each pixel until the pixel grid of each pixel is defined.

[0097] In this embodiment, by using the UV coordinates of each pixel as the center coordinates, the four neighboring coordinates in the horizontal and vertical directions are determined, and the center coordinates and neighboring coordinates are connected to form four spatial lines. The midpoints of each line are then located and connected sequentially to generate an initial grid, which is then rotated 90 degrees to obtain an orthogonal pixel grid. This embodiment constructs a pixel grid with directional consistency and structural symmetry for each pixel. By traversing all pixels, a systematic definition of the pixel grid in the entire image is achieved, providing a unified spatial reference for the alignment and stitching of RGB features and depth features.

[0098] For example, step S3 of the point cloud generation method specifically includes:

[0099] S3-1. Using UV coordinates as indices, anchor the target pixel in the four-modal tensor matrix;

[0100] S3-2. Based on K pre-trained convolutional kernels, perform convolution extraction on the four-modal tensors of the target pixel and its neighboring pixels to generate the convolutional feature vector of the target pixel.

[0101] Specifically, the convolutional feature vector represents a K-dimensional feature vector formed by K features generated at the UV coordinate position by performing local weighted summation on the four-modal tensors of the target pixel and its neighborhood using K pre-trained convolutional kernels respectively; wherein each feature corresponds to the mode response intensity of a convolutional kernel to the local region, and the feature vector integrates the semantic information of the target pixel under RGB intensity and depth.

[0102] S3-3. Decode the convolutional feature vector of the target pixel to obtain the binary classification probability that the target pixel belongs to the glue-coated area;

[0103] Specifically, the decoding means that the K-dimensional convolutional feature vector is mapped to a two-dimensional space through a fully connected layer, and then normalized by the Sigmoid activation function to output the probability value of the target pixel belonging to the category of the area to be coated, with the value range being [0,1].

[0104] S3-4. Traverse the UV coordinates until the binary classification probability of each pixel in the four-modal tensor matrix is ​​generated.

[0105] Specifically, the traversal of UV coordinates refers to starting from the top left corner of the image and sequentially performing steps S3-1 to S3-3 on each UV coordinate position in row priority order, that is, anchoring, extracting convolutional features and decoding vectors pixel by pixel, and finally generating pixel-level binary classification probabilities.

[0106] For example, step S4 of the point cloud generation method specifically includes:

[0107] S4-1. Compare the binary classification probability of each pixel in the four-modal tensor matrix with a set threshold.

[0108] S4-2. If the binary classification probability is not less than the set threshold, then mark the pixel as the point to be coated with glue; otherwise, mark it as the point not to be coated with glue.

[0109] S4-3. Traverse the four-modal tensor matrix and repeat S4-1 to S4-2 until you obtain the pixel mask map corresponding to H×W pixels.

[0110] Specifically, a pixel mask is an H×W matrix where each element corresponds to a pixel location in the original image, and the values ​​in the matrix are typically 0 or 1.

[0111] S4-4. In the pixel mask image, the pixel corresponding to the point to be coated with glue is marked as the target pixel.

[0112] S4-5. Traverse the pixel mask until J target pixels are determined.

[0113] In this embodiment, a pixel mask image is generated by comparing the binary classification probability with a set threshold, thereby determining J target pixels. These target pixels accurately identify the areas in the sole image that require adhesive application. Specifically, the selection of target pixels ensures that only those pixels identified as areas to be coated with adhesive are retained, eliminating interference from non-adhesive areas and avoiding misapplication or omissions.

[0114] Because this method is based on pixel-level classification, the obtained target pixels closely match the actual glue application boundary, which can truly reflect the glue application requirements under complex textures or depth variations. Especially when facing shoe sole surfaces with similar colors, uneven lighting, or visual camouflage, it can still stably extract effective areas to be glued.

[0115] For example, step S5 of the point cloud generation method specifically includes:

[0116] S5-1. Extract the UV coordinates of the target pixel and determine the pixel grid it belongs to based on the UV coordinates;

[0117] S5-2. Extract the depth features of the target pixel based on the pixel grid where the target pixel is located;

[0118] S5-3. Back-project the UV coordinates and depth features of the target pixel to calculate the point cloud coordinates of the target pixel;

[0119] Specifically, back projection means mapping the target pixel (with depth features) on a two-dimensional image plane back to its position in three-dimensional space, thereby realizing the transformation from the UV coordinates of the target pixel to the point cloud coordinates.

[0120] First, determine the UV coordinates of the target pixel and read the corresponding depth feature value from the depth image.

[0121] Then, the camera's intrinsic matrix is ​​used to convert the pixel coordinates (UV coordinates) and depth feature values ​​into 3D coordinates in the camera coordinate system. The camera intrinsic matrix contains information about the camera's internal parameters, such as focal length and principal point position, and is used to convert 2D image coordinates into 3D spatial coordinates. For example, the conversion formula is as follows:

[0122] ,

[0123] ,

[0124] ,

[0125] Where X, Y, and Z represent the 3D coordinates of the target pixel in the camera coordinate system, with Z being the depth feature value in the camera coordinate system; U and V are the UV coordinates of the target pixel. and These represent the focal lengths in the horizontal and vertical directions, respectively. and These represent the offsets of the image center point in the horizontal and vertical directions, respectively, while Depth represents the depth feature of the target pixel.

[0126] S5-4. Traverse the J target pixels and repeat S5-1 to S5-3 until the glue-coated area point cloud composed of J point cloud coordinates is generated.

[0127] In this embodiment, by extracting the UV coordinates of the target pixel and determining its corresponding pixel grid, and combining the depth features at that location, the two-dimensional image coordinates and depth values ​​are back-projected using the camera intrinsic parameter matrix to calculate the three-dimensional spatial coordinates of each target pixel in the camera coordinate system. Finally, all J target pixels are traversed to generate a point cloud of the adhesive application area composed of its three-dimensional coordinates. This method performs three-dimensional reconstruction based on the segmented pixels of the area to be adhesiveed, ensuring that the point cloud data covers the actual area to be adhesiveed and avoiding the introduction of invalid points. At the same time, since the back-projection process directly depends on the pixel-level aligned RGB features and depth features, the generated three-dimensional point cloud closely fits the geometric shape of the adhesive application boundary of the sole in spatial distribution. It can truly reflect the three-dimensional structure of complex curved surfaces, small gaps, or edge transitions, providing a clear and directly driveable geometric coordinate input for the robot's automated adhesive application, thus improving the efficiency of automated adhesive application for the sole.

[0128] Example 2: The technical solution of Example 2 differs from that of Example 1 in that it discloses the pre-training steps of the sole segmentation model in Example 1. These pre-training steps include:

[0129] A1. Obtain the set of camouflaged segmented images;

[0130] A2. Input the camouflaged segmentation image set into the convolutional neural network model for iterative supervised training to generate the shoe sole segmentation model.

[0131] Furthermore, step A1 in Example 2 specifically includes:

[0132] A1-1. Obtain RGBD images of several shoe sole samples;

[0133] A1-2. Perform binary classification labeling on each pixel point in the RGBD images of several shoe sole samples; the binary classification label includes: points to be coated with glue and points not coated with glue;

[0134] For example, in this embodiment, the labeling is completed using professional image annotation software.

[0135] A1-3. Perform image enhancement on the RGBD images of several shoe sole samples labeled with binary classification tags to generate several RGBD enhanced images;

[0136] It is important to note that the image enhancement described in this embodiment refers to operations performed without compromising or interfering with the accuracy of the original RGBD image and its corresponding binary classification labels (points to be coated and uncoated). These operations include color adjustment, noise addition, cropping, and padding. In short, during image enhancement, it is crucial to ensure that the enhancement operation does not increase the difficulty of object recognition in the original RGBD image or render the original binary classification labels invalid.

[0137] A1-4. Mix the RGBD images of several shoe sole samples labeled with binary classification with several RGBD enhanced images to generate a dummy segmentation image set.

[0138] In this embodiment, by acquiring RGBD images of several shoe sole samples and labeling each pixel with binary classification of glue-to-be-applied and glue-free points, and then enhancing the images, a semantically clear camouflage segmentation image set is constructed.

[0139] Furthermore, step A2 of embodiment 2 specifically includes:

[0140] A2-1. Input the camouflaged segmented image set into the convolutional neural network model;

[0141] A2-2. Extract the RGBD images of the current batch from the camouflaged segmented image set, and generate the pixel prediction mask map of the current batch through forward propagation.

[0142] Specifically, the pixel prediction mask is the result obtained by performing feature extraction and classification on the input RGBD image through a convolutional neural network model. Each pixel corresponds to a prediction probability value, which represents the probability that the pixel belongs to the point to be coated with glue.

[0143] A2-3. Calculate the cross-entropy loss between the pixel prediction mask of the current batch and the corresponding binary classification labels of the RGBD image;

[0144] Specifically, the cross-entropy loss is used to quantify the difference between the pixel prediction mask image of the current batch and the true binary classification label; the cross-entropy loss formula is as follows:

[0145] ;

[0146] Where L represents the cross-entropy loss of the current batch, P represents the total number of pixels in the current batch, including all pixels in the height and width directions, i.e., P=H×W; i represents the index of the pixel, from 1 to P; This represents the true binary label (0 or 1) of the i-th pixel in the current batch of RGBD images; where 1 represents a pixel to be coated with adhesive and 0 represents a pixel not coated with adhesive. This represents the predicted probability value of the i-th pixel in the pixel prediction mask that it belongs to the point to be coated with glue, and its range is between [0,1].

[0147] A2-4. With the goal of minimizing the cross-entropy loss, update the parameters of the convolutional neural network model, iteratively calculate the cross-entropy loss of the next batch, until the convolutional neural network model converges to the shoe sole segmentation model.

[0148] In this embodiment, a set of camouflaged segmented images is input into a convolutional neural network model. Forward propagation is performed to generate a predicted mask image for the current batch of pixels. The cross-entropy loss is calculated based on the difference between the predicted result and the true binary classification label. With the goal of minimizing this loss, the model parameters are iteratively updated through the backpropagation algorithm to gradually optimize the network's ability to distinguish between glue-coated and non-glue-coated points in the RGBD image, ultimately generating a shoe sole segmentation model with stable output.

[0149] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means.

[0150] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for generating point clouds of adhesive-coated areas on shoe soles based on RGBD camouflage instance segmentation, characterized in that, include: S1. Obtain the target shoe sole image; wherein, the target shoe sole image contains the RGB features and depth features of each pixel; S2. Construct the four-modal tensor matrix corresponding to each pixel on the target shoe sole image; The shape of the four-modal tensor matrix is ​​H×W, where H and W are the number of pixels in the height and width directions, respectively. The construction of the four-modal tensor matrix corresponding to each pixel on the target shoe sole image includes: On the target shoe sole image, construct UV coordinates for each pixel; Based on the UV coordinates constructed from each pixel, a pixel grid is defined for each pixel; The UV coordinates constructed based on each pixel define a pixel grid for each pixel, including: The UV coordinates of any pixel are anchored as the center coordinates; Using the central coordinates as the starting point, determine the coordinates of its four neighboring regions in the orthogonal direction; Connect the coordinates of the four neighboring regions with the center coordinates to form four spatial lines; Locate the four midpoints of the four spatial lines; Connect the four midpoints in a clockwise or counterclockwise order to form an initial grid centered on the center coordinates; The initial grid is rotated 90 degrees around the center coordinates to generate an orthogonal pixel grid. Iterate through the UV coordinates of each pixel until the pixel grid of each pixel is defined; S3. Input the four-modal tensor matrix into the pre-trained shoe sole segmentation model, and generate the binary classification probability of each pixel through forward propagation; S4. Determine J target pixels based on the binary classification probability of each pixel; S5. Generate a point cloud of the glued area of ​​the target shoe sole based on J target pixels; The step of generating a point cloud of the adhesive coating area of ​​the target shoe sole based on J target pixels includes: Extract the UV coordinates of the target pixel and determine the pixel grid it belongs to based on the UV coordinates; Extract the depth features of the target pixel based on the pixel grid where the target pixel is located; The UV coordinates and depth features of the target pixel are back-projected to calculate the point cloud coordinates of the target pixel. Traverse the J target pixels and repeatedly perform the calculation of the point cloud coordinates until a point cloud of the adhesive-coated area composed of J point cloud coordinates is generated.

2. The method for generating point clouds of the sole adhesive region based on RGBD camouflage instance segmentation according to claim 1, characterized in that, The construction of the four-modal tensor matrix corresponding to each pixel on the target shoe sole image includes: On the target shoe sole image, extract the RGB features and depth features corresponding to the pixel grid; By concatenating the RGB features and depth features located in the same pixel grid, a four-modal tensor corresponding to the pixel is generated; Iterate through each pixel of the target shoe sole image and repeatedly generate the four-modal tensor until H×W four-modal tensors are generated. The H×W four-modal tensors are arranged according to their spatial positions based on UV coordinates to construct a four-modal tensor matrix.

3. The method for generating point clouds of the sole adhesive region based on RGBD camouflage instance segmentation according to claim 1, characterized in that, The pre-training steps of the sole segmentation model include: A1. Obtain the set of camouflaged segmented images; A2. Input the camouflaged segmentation image set into the convolutional neural network model for iterative supervised training to generate the shoe sole segmentation model.

4. The method for generating point clouds of the sole adhesive region based on RGBD camouflage instance segmentation according to claim 3, characterized in that, The acquisition of the camouflaged segmentation image set includes: A1-1. Obtain RGBD images of several shoe sole samples; A1-2. Perform binary classification labeling on each pixel point in the RGBD images of several shoe sole samples; the binary classification label includes: points to be coated with glue and points not coated with glue; A1-3. Perform image enhancement on the RGBD images of several shoe sole samples labeled with binary classification tags to generate several RGBD enhanced images; A1-4. Mix the RGBD images of several shoe sole samples labeled with binary classification with several RGBD enhanced images to generate a dummy segmentation image set.

5. The method for generating point clouds of the sole adhesive region based on RGBD camouflage instance segmentation according to claim 4, characterized in that, The generation of the shoe sole segmentation model includes: A2-1. Input the camouflaged segmented image set into the convolutional neural network model; A2-2. Extract the RGBD images of the current batch from the camouflaged segmented image set, and generate the pixel prediction mask map of the current batch through forward propagation. A2-3. Calculate the cross-entropy loss between the pixel prediction mask of the current batch and the corresponding binary classification labels of the RGBD image; A2-4. With the goal of minimizing the cross-entropy loss, update the parameters of the convolutional neural network model, iteratively calculate the cross-entropy loss of the next batch, until the convolutional neural network model converges to the shoe sole segmentation model.

6. The method for generating point clouds of the adhesive coating area of ​​the shoe sole based on RGBD camouflage instance segmentation according to claim 1, characterized in that, The generation of binary classification probabilities for each pixel through forward propagation includes: S3-1. Using UV coordinates as indices, anchor the target pixel in the four-modal tensor matrix; S3-2. Based on K pre-trained convolutional kernels, perform convolution extraction on the four-modal tensors of the target pixel and its neighboring pixels to generate the convolutional feature vector of the target pixel. S3-3. Decode the convolutional feature vector of the target pixel to obtain the binary classification probability that the target pixel belongs to the glue-coated area; S3-4. Traverse the UV coordinates until the binary classification probability of each pixel in the four-modal tensor matrix is ​​generated.

7. The method for generating point clouds of the sole adhesive region based on RGBD camouflage instance segmentation according to claim 2, characterized in that, The step of determining J target pixels based on the binary classification probability of each pixel includes: S4-1. Compare the binary classification probability of each pixel in the four-modal tensor matrix with a set threshold. S4-2. If the binary classification probability is not less than the set threshold, then mark the pixel as the point to be coated with glue; otherwise, mark it as the point not to be coated with glue. S4-3. Traverse the four-modal tensor matrix and repeat S4-1 to S4-2 until you obtain the pixel mask map corresponding to H×W pixels. S4-4. In the pixel mask image, the pixel corresponding to the point to be coated with glue is marked as the target pixel. S4-5. Traverse the pixel mask until J target pixels are determined.

Images